Wearable electronic device displays a 3D zone from where binaural sound emanates

ABSTRACT

A portable electronic device (PED) divides an area around a user into a three-dimensional (3D) zone. A wearable electronic device worn on a head of the user displays the 3D zone in response to the wearable electronic device detecting that the user is leaving the zone. The wearable electronic device plays binaural sound that emanates to the user from sound localization points (SLPs) inside the zone.

BACKGROUND

People are able to localize binaural sound by sensing audio cues in theform of temporal and spectral differences heard between the left andright ears. These differences can be artificially created using HeadRelated Transfer Functions (HRTFs). HRTF's are individualized or uniquefunctions for each person since they depend on a size and a shape of aperson's head, face, ears, torso, and other physiological factors.

Unfortunately, it is difficult or burdensome to obtain the HRTFs for anindividual. Typically, the HRTFs of a person must be measured in ananechoic chamber or a specialized location that includes numerousspeakers, expensive sound equipment, and a soundproof environment.

Methods and apparatus that facilitate obtaining HRTFs and audio impulseresponses will advance technology that creates and maintains virtualenvironments, virtual reality, and augmented reality.

BRIEF DESCRIPTION OF THE DRAWINGS

FIG. 1 is a method to generate audio impulse responses from a soundgenerated or triggered with a hand-held portable electronic device(HPED) in accordance with an example embodiment.

FIG. 2 is a method to provide binaural sound to a person at a SLP thatis away from but proximate to the person in accordance with an exampleembodiment.

FIG. 3 is a method to capture impulse responses when an HPED is within acertain distance of a person in accordance with an example embodiment.

FIG. 4 is a method to determine a facial orientation of a listeningperson when an audio impulse response is recorded in accordance with anexample embodiment.

FIG. 5 is a method to designate one or more SLPs to a zone or areaaround a head of a person in accordance with an example embodiment.

FIG. 6 is a method to map a SLP to a location in accordance with anexample embodiment.

FIG. 7 is a method to select a SLP for a person and convolve sound tolocalize at the SLP in accordance with an example embodiment.

FIG. 8 is a method to retrieve RIRs and convolve sound based on alocation of a person in accordance with an example embodiment.

FIG. 9A is a top view of a person capturing audio impulse responses inaccordance with an example embodiment.

FIG. 9B is a back view of the person of FIG. 9A in accordance with anexample embodiment.

FIG. 9C is a side view of the person of FIG. 9A in accordance with anexample embodiment.

FIG. 9D is a top corner view of the person of FIG. 9A in accordance withan example embodiment.

FIG. 10A is a top view of a person capturing audio impulse responseswith a HPED attached to a pole in accordance with an example embodiment.

FIG. 10B is a front view of the person of FIG. 10A in accordance with anexample embodiment.

FIG. 10C is a side view of the person of FIG. 10A in accordance with anexample embodiment.

FIG. 1D is a top corner view of the person of FIG. 10A in accordancewith an example embodiment.

FIG. 11 shows a person sitting at a table and capturing audio impulseresponses with microphones in his left and right ears in accordance withan example embodiment.

FIG. 12 shows a person sitting in an automobile and capturing audioimpulse responses with microphones in his left and right ears inaccordance with an example embodiment.

FIG. 13 shows a person sitting in a chair at a table and beingsurrounded by a plurality of SLPs that are located away from butproximate to his face in accordance with an example embodiment.

FIG. 14 is an electronic system in accordance with an exampleembodiment.

FIG. 15 is another electronic system in accordance with an exampleembodiment.

FIG. 16 shows an electronic system that includes a HPED and a separatespeaker attached to a pole in accordance with an example embodiment.

SUMMARY OF THE INVENTION

One example embodiment is a method that captures audio impulse responsesusing electronic microphones in a left ear and a right ear of a person.A handheld portable electronic device (HPED) generates or triggers asound away from but proximate to the person. The microphones capture theaudio impulse responses that are used to convolve sound that localizesaway from but proximate to the person.

Other example embodiments are discussed herein.

DETAILED DESCRIPTION

Example embodiments relate to methods and apparatus that generate,manage, and perform tasks for audio impulse responses, including roomimpulse responses (RIRs), binaural room impulse responses (BRIRs),head-related impulse responses (HRIRs), and head-related transferfunctions (HRTFs).

As noted in the Background section, it is difficult or burdensome toobtain HRTFs of a person. Deriving a person's HRTFs with traditionalmethods can be time consuming and expensive. Further, facilities toperform such measurements are not abundant and generally not accessibleto the general public.

In some instances, HRTFs for a person are not derived from data takendirectly from the person but are derived from one or more approximationtechniques, such as obtaining HRTFs from a dummy head or approximatingHRTFs from HRTFs individualized for another person. These approximatedHRTFs are not as accurate as HRTFs measured directly from the person.

If the HRTFs are not sufficiently accurate for the person, then theperson may localize sounds to the wrong location, such as localizing asound toward the back of the head when the localization point is towardthe front of the head. Additionally, if the HRTFs are not accurate, thenthe listener can fail to localize sounds externally and instead hear thesounds as if they originate inside the head of the listener.

Example embodiments include systems, apparatus, and methods that capturehead related transfer functions (HRTFs) of a person and solve problemsassociated with binaural sound, including problems related to obtainingaccurate HRTFs for a person. With example embodiments, the HRTFs areindividualized or unique for the person and can be used to accuratelylocalize binaural sound.

By way of introduction, sound localization (i.e., the act of relatingattributes of the sound being heard by the listener to the location ofan auditory event) provides the listener with a three-dimensional (3D)soundscape or 3D sound environment where sounds can be externallylocalized to points around the listener, such as external positions towhich the listener can point. Binaural sound and some forms of stereosound provide a listener with the ability to localize sound; thoughbinaural sound generally provides a listener with a superior ability tolocalize sounds in 3D space.

Sound localization offers people a wealth of new technological avenuesto not only communicate with each other but also to communicate withelectronic devices, software programs, and processes. This technologyhas broad applicability in augmented reality (AR), virtual reality (VR),audio augmented reality (AAR), telecommunications and communications,entertainment, tools and services for security, medicine, disabledpersons, recording industries, education, natural language interfaces,and many other sectors.

As this technology develops, challenges will arise with regard to howsound localization integrates into the modern era. Example embodimentsoffer solutions to some of these challenges and assist in providingtechnological advancements in methods and apparatus using soundlocalization.

Binaural sound can be manufactured or recorded. When binaural sound isrecorded, two microphones are placed in or near human ears or placed inears of a dummy head. When this binaural recording is played back (e.g.,through headphones or earphones), audio cues in the recorded soundprovide the listener with an audio representation of the 3D space wherethe recording was made. The sound is quite realistic, and the listenercan localize sources of individual sounds with a high degree ofaccuracy.

Binaural sound typically delivers two types of localization cues:temporal cues and spectral cues. Temporal cues arise from an interauraltime difference (ITD) due to the distance between the ears. Spectralcues arise from an interaural level difference (ILD) or interauralintensity difference (IID) due to shadowing of sound around the head.Spatial cues are ITDs and ILDs and their combination.

A person hearing the spatial cues can localize sound or estimate alocation of a source of the sound. In some instances, a listener canexternalize and localize a source of binaural sound to a point andexperience the sound as indistinguishable from a real-world sound sourceoccurring in his physical environment. Processing sounds through alistener's individualized HRTFs provides a markedly higher degree ofrealism than using approximated or stock HRTFs.

Although stereo sound offers some degree of sound localization, stereosound and binaural sound are different. As explained in WIKIPEDIA, theterm “binaural sound” and “stereo sound” are frequently confused assynonyms. Conventional stereo recordings do not factor in natural earspacing or “head shadow” of the head and ears since these things happennaturally as a person listens and experiences his or her own ITDs(interaural time differences) and ILDs (interaural level differences).Headphones or earphones generally deliver binaural sound; although itcan be delivered with loudspeakers. Loudspeaker-crosstalk ofconventional stereo interferes with binaural reproduction, playbacksystems implement crosstalk cancellation to create a sweet spot forbinaural listening. As a general rule, binaural sound accommodates forone or more ITDs, ILDs, natural ear spacing, and head shadow. Theeffects of these on a source signal can be derived from an individual'sHRTF. Binaural sound can also be explained as causing or intending tocause one or more sound sources produced through headphones or earphonesas originating apart from but proximate to the listener.

Binaural sound spatialization can be reproduced to a listener usingheadphones or speakers, such as with dipole stereo (e.g., multiplespeakers that execute crosstalk cancellation). Generally, binauralplayback on earphones or a specially designed stereo system provides thelistener with a sound that spatially exceeds normally recorded stereosound since the binaural sound more accurately reproduces the naturalsound a user hears when at the location where the sound was recorded.Binaural recordings can convincingly reproduce the location of soundbehind, ahead, above, or any position the sound actually came fromduring recording.

Sound signals are modified as they travel from their original source andinteract with the human anatomy and surrounding environment. Thesemodifications encode the location of the original source and can becaptured as an impulse response. The impulse response for a human iscalled a head-related impulse response (HRIR), and it represents impulseresponses from a sound source to two ears in a free-field environment(without modification due to a room environment). A HRTF is a Fouriertransform of a HRIR.

A source sound can be convolved with a HRIR of a person. Convolvingsound in this manner joins the original sound with impulses responses sothe person hears the sound as if he or she were present at the sourcelocation when the sound was played. The HRIRs describe how to alter thesound source before the sound is provided to the ears of the listener.

Impulse responses for a room or a particular location are room impulseresponses or RIRs, whereas impulse responses for a room as experiencedby a particular person in the room with two ears are binaural roomimpulse responses or BRIRs. For example, BRIRs characterize the transferof sound from a source location in a room to the entrances of the leftand right ears of a listener (such as a person or a dummy-head). BRIRscan be obtained by measuring RIRs and HRIRs separately orsimultaneously. Further, BRIRs, HRIRs, and RIRs can be obtained withoutmeasuring, such as being generated from computer modeling of impulseresponses. As one example, RIRs and HRIRs are measured with microphonesat one or more locations. As another example, RIRs and HRIRs aregenerated with one or more computer models. As yet another example,HRIRs are measured with microphones; RIRs are generated with a computermodel; and HRIRs and RIRs are combined to generate BRIRs. As yet anotherexample, individualized or customized HRIRs and/or HRTFs are generatedfrom measurements and/or calculations based on an anthropometry-basedmethod or subjective-selection-based method (such as customizing HRTFsfor a person based on anatomical feature similarities of an individualwith known, measured HRTFs of another individual). Transfer functionsand impulse responses can also be generated from one or moreinterpolation techniques (such as interpolating a HRTF at a locationbetween two known HRTFs or using a nearest known or measured location).Furthermore, impulse responses can be extracted to generate transferfunctions, such as removing RIRs from a set of BRIRs to yieldHRIRs/HRTFs or removing impulse responses due to a person's outstretchedarm while holding the HPED to generate the sound.

FIG. 1 is a method to generate audio impulse responses from a soundgenerated or triggered with a hand-held portable electronic device(HPED).

Block 100 states generate, from a speaker of a HPED, a sound while theHPED is at a location that is away from but proximate to a face of aperson wearing earphones with microphones.

Each ear of the person includes a microphone (i.e., the left ear has amicrophone, and the right ear has a microphone). The earphones caninclude one or more of microphones, speakers, volume control, an on/offswitch or power button, wireless communication circuitry, noisecancellation circuitry, etc. For example, the earphones include a leftearphone with a left microphone and a left speaker and also include aright earphone with a right microphone and a right speaker.

The earphones can directly connect to the HPED or wirelessly communicatewith the HPED and/or a wireless network. For example, the earphonestransmit and receive data over the Internet through a wireless network.

Further, example embodiments are not limited to earphones but includeother electronic devices with microphones that can be positioned in ornear an ear of a person or dummy or otherwise record impulse responses.Examples of such devices with microphones include, but are not limitedto, headphones, wearable electronic glasses, optical head mounteddisplays (OHMDs), heads-up displays, and other electronic devices withmicrophones that can record impulse responses or sound.

The speaker of the HPED plays or generates a sound that is used todetermine audio impulse responses of the person and/or the room orenvironment where the person is located. By way of example, theseimpulse responses include one or more of HRIRs, BRIRs, and room impulseresponses (RIRs).

Further, example embodiments are not limited to a HPED that generatesthe sound but include other devices that can generate a sound to recordaudio impulse responses for a person. For example, the HPED communicateswith and/or triggers another device to play sound. Examples of devicesthat can generate a sound include, but are not limited to, wearableelectronic devices, televisions, desktop computers, speakers, and otherelectronic devices that generate sound. Further, such sound generatingdevices can be non-electronic devices such as clickers, and includepeople (such as a person generating a sound by clapping, snappingfingers, knocking, or talking).

In an example embodiment, the sound generates left and right audioimpulse responses that can be used to generate individualized oruser-specific sets of HRTFs, HRIRs, or BRIRs. Different types of soundscan be used to generate these audio impulse responses. By way ofexample, such sounds include, but are not limited to, a known spectrumstimulus sound, a frequency-swept sine wave, a click, a pulse, a maximumlength sequence (MLS), a pseudo-random binary sequence, a ping, acomplementary Golay code, a voice announcing a word or a phrase, oranother type of sound.

The HPED generates or triggers the sound while the HPED is away from butproximate to the face of the person wearing earphones with microphones(or other electronic device as noted herein). A distance from the faceof the person to the HPED can range from a few inches to several meters.Preferably, this distance is sufficient so audio impulse responses canbe generated from the sound and used to subsequently convolve sound soit externally localizes to the listener. For example, the HPED ispositioned away from the person at a distance and angle such that thesound it generates provides microphones in a person's ears withsufficient impulse responses to generate a set of HRTFs, HRIRs, or BRIRsfor the person. When this set is used to convolve a voice signal, theperson localizes the voice to a point proximate but away from him, suchas localizing sound to a SLP in empty space or localizing sound to a SLPon an object.

A HRTF is a function of frequency (f) and three spatial variables (r, θ,ϕ) in a spherical coordinate system. Here, r is the radial distance froman origin of the sound to the recording point; θ (theta) is the azimuthangle between the origin and recording point; and ϕ (phi) is the polarangle, elevation, or elevation angle between the origin and recordingpoint.

When the distance (r) is greater than or equal to about one meter (1 m)as measured from the sound source to the capture point (e.g., the headof the person), the sound attenuates inversely with the distance. Onemeter or thereabout defines a boundary between near field and far fieldHRTFs. A “near field” HRTF is measured from about one meter or less;whereas a “far field” HRTF is measured from about one meter or more.Example embodiments can be implemented with near field and far fielddistances.

Block 110 states capture, with a left microphone of the earphones in aleft ear of the person and with a right microphone of the earphones in aright ear of the person, the sound generated from the speaker of theHPED.

The left microphone captures the sound at the left ear, and the rightmicrophone captures the sound at the right ear. Further, as noted, theearphones can be electrically connected to and/or in communication withan electronic device, such as being physically coupled or connected to asound jack on the HPED or being in wireless communication with the HPED,a network, a server, or another electronic device.

Block 120 states generate, from the sound captured in the left and rightmicrophones, a set of audio impulse responses or audio transferfunctions for the person.

When the microphones are positioned in the ears of the listening person,the HPED generates the sound while being at a distance (r) from the headof the person, at an azimuth angle (θ), and at an elevation angle (ϕ)(i.e., being at (r, θ, ϕ)). The captured sound is processed to generateone or more sets of HRTFs, HRIRs, RIRs, and BRIRs for the person and/orthe location.

During the processing, aspects of the measured impulse responses can beremoved. For example, impulse responses attributed to the earphones(known as common transfer functions or CTFs) to yield a head relatedtransfer function (HRTF) or directional transfer function (DTF) for thelocation from where the sound was generated. CTFs are also known asheadphone or earphone transfer functions and can also be measured andappropriately inverted.

Consider an example for calculating a HRTF or DTF for a known audiosignal s(t) that generates from a speaker of a HPED at a position(distance, azimuth, elevation) or (r, θ, ϕ). The calculations below arefor the left (l) ear, but are equally applicable to the right ear:m _(l,θ,ϕ)(t)=s(t)·c(t)·d _(l,θ,ϕ)(t);M _(l,θ,ϕ)(ω)=S(ω)C(ω)D _(l,θ,ϕ)(ω).

Here, m_(l,θ,ϕ)(t) is the signal recorded with the microphone of theleft ear given the original known sound, S(t); C(t) is the known commontransfer function or CTF; and d_(l,θ,ϕ)(t) is the unknown directionaltransfer function for the left ear. As such, the signal recorded at theleft ear is a function of the known original signal, the CTF, and theunknown directional transfer function at the (r, θ, ϕ), with thedistance (r) being one meter or greater or measured for a far fieldHRTF.

The above equations are rewritten to solve for d_(l,θ,ϕ)(t) as follows:|D _(l,θ,ϕ)(ω)|=|M _(l,θ,ϕ)(ω)|/((|S(ω)|)(|C(ω)|));∠D _(l,θ,ϕ)(ω)=∠M _(l,θ,ϕ)((ω)−∠S(ω)−∠C(ω);D _(l,θ,ϕ)(ω)=|D _(l,θ,ϕ)|(ω)exp(j∠D _(l,θ,ϕ)(ω));d _(l,θ,ϕ)(t)=F ⁻¹(D _(l,θ,ϕ)(ω)).

The corresponding directional transfer function for the right ear wouldthus be:d _(r,θ,ϕ)(t)=F ⁻¹(D _(r,θ,ϕ)(ω)).

Following the derivation, the HRTFs can be stored in the AES69 spatialacoustic data file format.

FIG. 2 is a method to provide binaural sound to a person at a SLP thatis away from but proximate to the person.

Block 200 states obtain sound to provide to a person wearing earphones.

One or more electronic devices capture or provide the sound. The soundcan be delivered over a wired or wireless network, from a server, orfrom a computer or local electronic device. Some example sources of thesound include, but are not limited to, sound streamed and provided inreal-time from a telephony application or a live video call, soundprovided by a computer program such as a multiplayer game with voices ofother people, characters, and sound effects, sound output by a computerprogram with a voice interface, sounds played from a sound or musiclibrary or database, or sound from all of these sources or othersources.

The person can wear various types of electronic devices that providebinaural sound. Examples of such electronic devices include, but are notlimited to, earphones, headphones, electronic glasses, a head mounteddisplay, a heads-up display, or another wearable electronic device (suchas a device with two or more speakers). Furthermore, binaural sound canbe provided to a person that does not wear earphones, such as providingbinaural sound through two or more speakers.

Block 210 states convolve the sound with a set of impulse responses ortransfer functions for the person.

Sound can be convolved either directly in the time domain with a finiteimpulse response (FIR) filter or with a Fast Fourier Transform (FFT).For example, an electronic device convolves the sound with a set ofHRTFs, HRIRs, BRIRs, or RIRs and provides the person with binauralsound.

In an example embodiment, convolution involves an audio input signal andan impulse response. The input signal can be a limited length audiosignal (such as a pre-recorded digital audio file) or an ongoing audiosignal (such as sound from a microphone or streaming audio over theInternet from a continuous source). The impulse response can be a set ofHRIRs, BRIRs, RIRs, etc.

Convolution applies one or more FIR filters to the input signals andconvolves them into binaural audio output or binaural stereo tracks,such as convolving the input signal into binaural audio output that isspecific or individualized for the listener based on one or more of thelistener's impulse responses.

The FIR filters are derived binaural impulse responses that are obtainedfrom example embodiments discussed herein (e.g., derived from signalsreceived through microphones placed in, at, or near the left and rightear channel entrance of the person). Alternatively or additionally, theFIR filters are obtained from another source, such as generated from acomputer simulation or estimation, generated from a dummy head,retrieved from storage, etc. Further, convolution of an input signalinto binaural output can include sound with one or more ofreverberation, single echoes, frequency coloring, and spatialimpression.

Processing of the sound can also include calculating and/or adjusting aninteraural time difference (ITD), an interaural level difference (ILD),and other aspects of the sound in order to alter the cues andartificially alter the point of localization. Consider an example inwhich the ITD is calculated for a location (θ, ϕ) with the time-domainDTFs calculated for the left and right ears per the equations above. TheITD is located at the point for which the function attains its maximumvalue, known as the argument of the maximum or arg max as follows:

${ITD} = {\arg\mspace{14mu}{\max(\tau)}{\sum\limits_{n}{{d_{l,\theta,\phi}(n)} \cdot {{d_{r,\theta,\phi}\left( {n + \tau} \right)}.}}}}$

Subsequent sounds are filtered with the left HRTF, right HRTF, and ITDso that the sound localizes at (r, θ, ϕ). Such sounds include filteringstereo and monaural sound to localize at (r, θ, ϕ). For example, givenan input signal as a monaural sound signal s(n), this sound is convolvedto appear at (θ, ϕ) when the left ear is presented with:s _(l)(n)=s(n−ITD)·d _(l,θ,ϕ)(n);and the right ear is presented with:s _(r)(n)=s(n)·d _(r,θ,ϕ)(n).

Block 220 states provide, through the earphones worn by the person,binaural sound such that the binaural sound localizes to the person at asound localization point that is away from but proximate to the person.

After the input signal is convolved, it can be provided to the person(listener), stored, transmitted, further processed, etc. Although someexample embodiments teach that the sound is provided through earphones,binaural sound can also be provided to a person through two or moreloudspeakers, such as through stereo speakers positioned in a room withlistener or through car speakers.

Consider an example in which a dedicated digital signal processor (DSP)executes frequency domain processing to generate real-time convolutionof monophonic sound to binaural sound.

By way of example, a continuous audio input signal x(t) is convolvedwith a linear filter of an impulse response h(t) to generate an outputsignal y(t) as follows:

y(τ) = x(τ) ⋅ h(τ) = ∫₀^(∞)x(τ − t) ⋅ h(t) ⋅ dt.

This reduces to a summation when the impulse response has a given lengthN and the input signal and the impulse response are sampled at t=iDt asfollows:

${y(i)} = {\sum\limits_{j = 0}^{N - 1}{{x\left( {i - j} \right)} \cdot {{h(j)}.}}}$

Execution time of convolution further reduces with a Fast FourierTransform (FFT) algorithm and/or Inverse Fast Fourier Transform (IFFT)algorithm.

Consider another example of binaural synthesis in which recorded orsynthesized sound is filtered with a binaural impulse response (e.g.,HRIR or BRIR) to generate a binaural output sound to the person. Theinput sound is preprocessed to generate left and right audio streamsthat are mapped to one or more virtual sound sources or soundlocalization points (known as SLPs). These streams are convolved with abinaural impulse response for the left ear and the right ear to generatethe left and right binaural output sound signal. The output sound signalcan be further processed depending on its final destination, such asapplying a cross-talk cancellation algorithm to the output sound signalwhen it will be provided through loudspeakers or applying artificialbinaural reverberation to provide 3D spatial context to the sound.

One problem is that a distance from the source of the sound (e.g., thespeaker in a HPED) and the recording location, (e.g., the head of theperson wearing microphones in both ears) can impact the quality ofimpulse responses captured and subsequently used to convolve sound forthe listener. For example, if this distance is too close (e.g., underone meter), then near-field HRTFs will be captured. When the distance isgreater than about one meter, then far-field HRTFs will be captured. Ifthis far-field distance is too far, however, the arriving sound signalmay be too weak to effectively capture impulse responses at the head ofthe listener. Additionally, the sound arriving at the microphones may beoverly affected with sound reverberations from the room or locationwhere the recordings occur. Further, a person attempting to captureimpulse responses to generate binaural sound may be unable to designateor measure optimal locations or distances for placing the HPED withrespect to the head of the person.

Example embodiments, including FIG. 3 and other figures, address theseproblems and others.

FIG. 3 is a method to capture impulse responses when an HPED is within acertain distance of a person.

Block 300 states determine a distance from a HPED to a face of a personwhen the HPED is away from but proximate to the person.

A determination is made of the distance between the HPED and the face ofthe person in order to know the distance when the HPED generates thesound to capture the impulse responses. By way of example, the HPED oranother electronic device can determine this distance with variousapparatus and methods that include, but are not limited to, a camera(such as measuring distance based on one or more pictures or video), alight emitter and sensor (such as a laser or infrared emitter anddetector), ultrasonic range finder, and a proximity sensor.Alternatively, the distance can be measured (such as measured with atape measure). As another example, the distance can be measured orapproximated with a device, such as attaching the HPED to a pole,monopod, tripod, rod, or selfie-stick that has a known length and/orheight.

Block 310 makes a determination as to whether the distance is within arange to capture impulse responses of the person.

The range depends on various factors, such as a quality of soundgenerated from or triggered by the HPED, the quality and type ofmicrophones used, an amount of ambient noise present, an amount ofreverberation or attenuation, a type of room, a type of communicationfor which the impulse responses will be used, and the type of impulseresponses that a user or electronic device desires to capture.

For near-field impulse responses, the range is from 0.0 meters to about1.0 meter. For far-field impulse responses, the range is from about 1.0meter to about 2.5 meters. These values are exemplary since the rangescan be further divided. For example, the far-field range can beshortened or lengthened depending on a type of sound being captured,reverberation at the location, an amount of reverberation desired, atype of communication for which the impulse response will be used, andother factors.

As noted, one factor is the type of communication for which the impulseresponse will be used. For example, if an impulse response is beingcaptured for voice telephony, the user may desire to have such impulseresponses captured within a specific or predetermined range, such as1.0-1.5 meters. As another example, if the impulse response is beingcaptured for an advertisement, then the listener may not want theadvertisement to localize close to the head of the listener. As such,the range for advertisements can be farther, such as 1.5-2.0 meters. Asyet another example, a listener may want voices of an intelligent useragent (IUA) or an intelligent personal assistant (IPA) to localizerelatively closer to his or her head. As such, the range for thesevoices can be relatively closer, such as 0.5-1.1 meters.

If the answer to this determination is “no” then flow proceeds to block320 that states take action.

An action occurs when the distance is not within a specified orpredetermined range. For example, the distance is not in an optimalrange to generate useful impulse responses.

Example actions include, but are not limited to, providing the user orother person with an audible warning (e.g., playing a sound warning fromthe HPED), providing the user or other person with a visual warning(e.g., displaying a visual indication on the HPED or other display, suchas a text or light or an image), preventing the HPED from generating thesound to capture the impulse response (e.g., the HPED will notautomatically generate the sound), allowing the HPED to generate thesound (e.g., the HPED generates the sound but notes a warning ordesignation with the distance), instructing the user or other person tochange the distance (e.g., providing a written message or announcementfrom a voice to move closer or farther away in order to be within therange), capturing multiple impulse responses from the same location orother locations proximate to the location (e.g., capturing more impulseresponses when the listener is not within a range), processing theimpulse responses to compensate for the distance, or taking anotheraction.

If the answer to the determination in block 310 is “yes” then flowproceeds to block 330 that states generate a sound from the HPED tocreate the impulse responses of the person.

The HPED generates a sound so the microphones can capture an impulseresponse that is used to convolve sound for the listener. The HPED canalso trigger or cause another device to generate a sound. For example,the HPED communicates with another HPED or speakers proximate to theperson with the microphones, and this communication causes the otherspeakers to generate a sound to capture an impulse response. Forexample, the HPED instructs a stereo system to generate the soundthrough stereo speakers in a home theater system, or through a Bluetoothspeaker mounted on a hand-held pole or another location around thelistener such as a table.

Block 340 states generate the impulse responses and/or transferfunctions of the person based on the sounds captured in the left andright microphones of the person.

The microphones capture the sounds at the left and right ears, and thesesounds generate the set of HRTFs, HRIRs, RIRs, and/or BRIRs for thelocation of the HPED relative to the person. Subsequent sounds that areconvolved with this set of impulse responses or transfer functions willlocalize to the listener at the position of the HPED when the HPEDgenerated the sound for the impulse responses.

The sounds captured at the microphones can be further processed togenerate specific impulse responses or transfer functions. For example,the sounds are processed to remove CTFs, such as those associated withthe earphones or those associated with an arm of a person (e.g., in asituation in which a person holds the HPED away from his or her face andgenerates the sound). The sounds can also be processed to remove impulseresponses associated with the room in which the listener and HPED arelocated (such as removing RIRs from the impulse responses to generateHRIRs).

Block 350 states store the distance when the HPED generates the sound.

The distance between the HPED (or sound generating device) and thelistening person wearing the microphones is recorded, stored,transmitted, processed, etc. For example, each set of impulse responses(i.e., one for the left ear and one for the right ear) has an associateddistance from the listening person.

Block 360 states determine and store a location of the HPED and/or theperson when the HPED generates the sound.

A record is created or updated that contains the SLP that localizessound for the listener at the point where the sound was emitted. Therecord also contains the associated impulse responses and transferfunctions, an identifier of the user whose head responses were measured,details of the capture process and context, and other information, suchas other information discussed herein.

A HPED or other electronic device can determine its location withvarious apparatus and methods that include, but are not limited to,Global Positioning System or GPS (including assisted and synthetic GPS),cellular identification, WiFi (including received signal strengthindication and wireless fingerprinting), internal sensors (including acompass, a magnetometer, an accelerometer, a gyroscope, or otherinertial sensors), an ultrasonic or short-range wireless systems(including radio-frequency identification or RFID, near-fieldcommunication or NFC, broadcasting and reception of ultrasonic tones,Bluetooth beacons, and local transmitters (including transmittersmounted on buildings)), a camera (including a camera in a HPED), andcombinations of these methods and apparatus.

The location and orientation of the HPED (or sound generating device)and the listening person wearing the microphones are recorded, stored,transmitted, processed, etc. For example, each set of impulse responses(i.e., one for the left ear and one for the right ear) has one or moreassociated locations (such as having a location for the listening personand/or a location for the HPED or sound generating electronic device).

The distance, the location, and the orientation are stored for eachimpulse response. For example, this information includes (r, θ, ϕ). Thislocation can also include a GPS location or other location informationthat identifies where the HPED and/or listening person were when theimpulse responses were generated and recorded.

If the impulse responses are not recorded in an anechoic chamber orspecial sound room or location, then each location is unique and willinclude room impulse responses (RIRs). As such, the sound recorded withthe microphones can be processed to include BRIRs since the impulseresponses include both HRIRs of the person and RIRs of the surroundingenvironment.

Locations are unique and so are the BRIRs captured for each person ateach different location. An electronic device stores these locations andthe associated impulse responses so desired impulse responses can besubsequently retrieved for use at the same location, for use in asimilar location, or for use according to the wish of the user withoutregard to his location.

Consider an example in which a person desires to localize voicetelephony to a location three feet from his head at either forty-fivedegrees to his right or forty-five degrees to his left. One SLP is (3.0feet, +45°, 0°) and is stored as SLP1; and another SLP is (3.0 feet,−45°, 0°) and is stored as SLP2. The person captures BRIRs at threedifferent environmental locations for these two SLPs. Theseenvironmental locations include BRIRs captured at his office (stored as“Office”), BRIRs captured at his house (stored as “House”), and BRIRscaptured at the park (stored as “Park”). The person further designatesindividuals stored in his electronic contact list to both a SLP and anenvironmental location. For example, the person designates his co-workerAlice as Office SLP1, and designates his wife as House SLP2. When Alicecalls, her voice localizes at (3.0 feet, +45°, 0°) with reverberationsfrom the office. In other words, her voice sounds like she is at theoffice. When his wife calls, her voice localizes at (3.0 feet, −45°, 0°)with reverberations from their home. In other words, her voice soundslike she is at the house.

Traditionally, impulse responses were captured in a strictly controlledenvironment in which microphones were placed in ears of a real person ora dummy-head while the person or dummy-head was placed in an anechoicchamber and surrounded by many speakers. The distances between thespeakers and person were known, and the azimuth and elevation anglesfrom the face of the person or recording point to the speakers were alsoknown. In this manner, HRTFs could be captured for precise locationsaround the head of the person or dummy.

One problem is the general public, however, does not have convenient oreasy access to an anechoic chamber or controlled sound environment withexpensive sound equipment.

Example embodiments, including FIG. 4 and other figures, solve this andother technical problems associated with generating, capturing, anddetermining individualized impulse responses for people. Exampleembodiments capture impulse responses without this type of controlled orexpensive environment. Instead, people can capture their individualizedHRTFs, HRIRs, and/or BRIRs without relying on the expensive andtime-consuming traditional practice of capturing impulse responses in ananechoic chamber or other specialized sound location.

FIG. 4 is a method to determine a facial orientation of a listeningperson when an audio impulse response is recorded.

When an impulse response is captured and recorded at microphones locatedin the ears of the listening person, a facial orientation or headorientation of the person is determined and recorded. This facial orhead orientation can include one or more of an azimuth angle of theface, an elevation angle of the face, a tilt of the face, a generallocation of the direction of gaze with respect to another object (suchas a HPED), and a facial expression or emotion of the face. By way ofexample, a facial or head orientation can be measured and recorded withrespect to yaw, pitch, and roll of the head of the person. As anotherexample, facial orientation can be measured, described, and/or storedwith respect to a HPED. For instance, the HPED is located on a rightside or left side of a face of a person, and the facial orientation isdetermined with respect to this location of the HPED.

Block 400 states determine a facial orientation of a person with respectto a HPED when the HPED is away from but proximate to the person.

Two factors determine a gaze of a person: a head or facial orientation(i.e., face pose or face direction) and eye orientation (i.e., eye gazedirection). Typically, the facial orientation determines a globaldirection of the gaze, and the eye orientation determines a localdirection of the gaze. When a head of the person is level and he or shelooks straight ahead, then the line-of-sight of the eye gaze and thefacial orientation are straight ahead with 0° azimuth and 0° elevationand 0° tilt. This head position can also be described as a neutral headorientation position or neutral facial orientation with an X-Y-Zcoordinate system or yaw, pitch, and roll to be (0°, 0°, 0°).

Various methods and/or apparatus can measure, determine, or estimate adeviation or variance from this neutral facial orientation and/or obtainone or more measurements of the yaw, pitch, and roll of the head orazimuth and elevation angles. One example embodiment uses facialrecognition to determine or estimate a facial or head orientation of theperson from one or more images or video captured with a camera in theHPED. The facial orientation can be described or recorded with respectto a location of the HPED that is generating the sound for the impulseresponses.

One way to determine facial orientation is with a video-based facialorientation determination. This includes head-based approaches,ocular-based approaches (or eye-based approaches), and combinations ofthese two approaches.

Another way to determine facial orientation is with aclassification-based facial orientation determination. This determines arelationship between face pose and its appearance via a statisticallearning algorithm. Facial orientation is determined from face samplesand various factors, such as illumination, pose variation, expression,etc.

Another way to determine facial orientation is with a geometry-basedfacial orientation determination. This builds a three-dimensional (3D)model of the face to determine facial orientation. For instance, facialcontour and facial components of the person are matched with theirprojection on a 2D image.

Facial orientation can be determined from the relative position ofsalient anatomical local facial features of the person (such as arelative position of the nose, eyes, lips, and ears of the person). Forexample, the face is partitioned into several regions by detectingpixels that correspond to one or more salient facial features, facialcomponents, or facial regions. Facial orientation is estimated from therelative positions of the salient regions.

Consider an example in which the HPED includes or communicates with areal-time facial orientation determiner, such as a real-time faceinterpretation engine for smartphones. This determiner estimates facialorientation or head poses (e.g., pitch, roll, and yaw) from facialimages captured with the camera in the smartphone and from one or moremotion sensors in the smartphone. For instance, an accelerometer andgyroscope in the smartphone detect tilt and motion of the smartphone,and the camera captures images or video of the person. Readings from theaccelerometer and gyroscope provide information with regard to thedirection of gravity and an intensity of the motion of the smartphone.As such, the smartphone is not required to be held in a particularorientation to determine the facial orientation of the person at whomthe camera is directed. By way of example, an AdaBoost object detectionalgorithm detects a location of the face on the display of thesmartphone. A spatial relationship between feature points or edges orcorners of these feature points on the face (or landmarks) provides anestimation of facial orientation of the person.

Consider an example in which a camera in a HPED (such as a smartphone)captures an image of the person, and facial recognition softwaredetermines 2D feature points on the image and reconstructs a 3D pose. Analgorithm (such as Pose from Orthography and Scaling with Iterations orPOSIT) estimates the facial orientation of the person.

Consider an example in which a facial interpretation engine thatexecutes on a smartphone estimates or determines facial orientation of aperson.

Facial orientation can be determined with or without the use of a cameraor image of the person. For example, the sound generated from the HPEDand provided to the microphones positioned in the ears of the personprovides information as to the facial orientation of the person. Whenthe HPED is positioned directly in front of the face of the person, theITD between the left and the right ear is zero. The ITD between the twoears, however, changes in a predictable or known amount as the azimuthangle increases or decreases. A measurement of the ITD thus correlatesto a facial orientation of the person.

Facial orientation can also be determined with a compass. For example,the HPED includes a compass that measures and records a direction of aforward-looking direction of the face. Thereafter, the azimuth angle iscalculated from compass directions of the HPED as it points toward andmoves around a head of the person. Consider an example in which theperson looks north at a compass heading of 0°. The HPED is positioned1.0 meter from the face of the person and the camera lens of the HPEDpoints at the head to a direction of south or 180°. Here, the HPED isdirectly in front of the person at an azimuth of 0°. The HPED then movesalong an arc (i.e., maintaining its distance of 1.0 meter) until itscompass points to South West or 225° while continuing to point the lensto the head of the person. The azimuth angle with respect to the personis calculated by subtracting the current compass direction of 225° from180° to yield 45°, which corresponds to the facial orientation of theperson with respect to the current location of the HPED.

A person can also determine or assist in determining facial orientation.For example, an HPED instructs a person to position the HPEDapproximately three to four feet from the face of the person at anazimuth angle of about 20°-45°. When the person confirms the designatedplacement, the HPED generates the sound to capture the impulse responsesin the microphones located in the ears of the person. This locationwhere the sound was generated relative to the face of the personrepresents a SLP (i.e., a point or area where sound is localized).Thereafter, the HPED convolves sound so it localizes to this relativelocation for the person. The HPED can also use a timer or time delayfeature to allow a person to return to his designated position prior tothe sound being emitted.

Facial orientation and/or the location of the HPED can be based on areference point, such as a point in a spherical coordinate system, apoint in the X-Y-Z coordinate system, or another point. For example, asmartphone captures an image of a person while the smartphone is locatedone meter away from the face of the person and at a left side of theperson. The smartphone determines the facial orientation to be lookingto the right with respect to the location of the smartphone, and recordsits location as being away from the face and on its left side. Anexample embodiment can use a combination of two or more of these methodsfor a higher probability of accuracy.

Block 410 makes a determination as to whether the facial orientation iscorrect.

If the answer to this determination is “no” then flow proceeds to block420 that states take an action.

An action occurs when the facial orientation is not correct or notpreferred. For example, the person may be facing or looking in the wrongdirection. As another example, the HPED or speaker of the HPED may befacing or pointing in a wrong direction. As yet another example, a SLPfor the current facial orientation is already captured. As anotherexample, the facial orientation may be correct, but the HPED is tooclose to the person, too far away from the person, or at an incorrectelevation angle with respect to the person.

Example actions include, but are not limited to, providing the user orother person with a sound warning (e.g., playing a sound warning fromthe HPED), providing the user or other person with a visual warning(e.g., displaying a visual indication on the HPED, such as a text orlight or an image), preventing the HPED from generating the sound tocapture the impulse response (e.g., the HPED will not automaticallygenerate the sound), allowing the HPED to generate the sound (e.g., theHPED generates the sound but notes a warning or designation with thefacial orientation), instructing the user or other person to change thefacial orientation or the orientation of the HPED (e.g., providing awritten message or announcement from a voice to move or rotate the HPEDand/or change a head orientation of the person), capturing multipleimpulse responses from the same location or other locations proximate tothe location (e.g., capturing more impulse responses when the listenerand/or HPED does not have a specified or correct orientation),processing the impulse responses to compensate for the facialorientation, or taking another action.

If the answer to this determination is “yes” then flow proceeds to block430 that states generate a sound from the HPED to capture audio impulseresponses of the person.

The HPED generates the sound, and the microphones at the ears of thelistener capture the impulse responses at the listener.

Correct facial orientation can depend on one or more factors including,but not limited to, current settings of the HPED, preferences of theperson, desired locations for SLPs, previous SLP or impulse responsescaptured, environmental conditions, accuracy of determining orestimating impulse responses, a location of the HPED with respect to theface of the person, a location of the person and/or HPED, a level oramount of background noise, an orientation or rotation of the HPED withrespect to the head orientation of the person, what sound is selected togenerate the impulse responses, a distance between the face of theperson and the HPED, an intended use for the impulse responses (e.g.,used to localize sound in gaming applications, telephony applications,intelligent user agent or intelligent personal assistant applications,etc.), physical attributes of the listening person (such as his or herage, size, hair, amount of face exposed, amount of ears exposed, etc.),and other factors.

Further, the facial orientation can be calculated and stored withspecific coordinates or locations (such as a specific azimuth and/orelevation angle), and/or other coordinates (such as an (x, y, z)position and orientation (yaw, pitch, roll) in the room), and/or withgeneral coordinates or locations (such as located at a right side of aface, located at a left side of a face, located above a head of theperson, etc.).

Block 440 states store the facial orientation of the person and/or theorientation of the HPED when the HPED generates the sound.

In addition to storing an orientation of the face and/or HPED, otherinformation can be determined and stored as well. By way of example,this other information includes, but is not limited to, a height of theHPED from ground, altitude above sea level, a height of a face of theperson from ground, objects recognized between the person and the HPED,objects proximate to the person, a room scan and the positions of theHPED and listener within the room, a distance between the HPED and theface of the person, a time of day and calendar date, a location of theHPED and/or person when the impulse responses are captured, a decibellevel of the sound generated from the HPED, a type of sound used orselected to generate the impulse responses, a number of impulseresponses generated and captured, ambient sound or background at thelocation of the HPED and/or person when the impulse responses arecaptured, environmental conditions (such as temperature, humidity, etc.)when the impulse responses are generated and captured, and other datadiscussed herein.

Consider an example in which a smartphone executes an application thatassists a user (Alice) in automatically obtaining impulse responses soshe can enjoy localized voices for voice telephony. Alice desires tolocalize voices for calls in one of three locations or at one of threeSLPs: SLP 1 defined as stereo sound (where voices appear to originateinside her head), SLP 2 defined as approximately three feet from a rightside of her face at an azimuth angle of about 20° to 45° and anelevation angle of about 0°, and SLP 3 defined as approximately threefeet from a left side of her face at an azimuth angle of about negative20° to negative 45° and an elevation angle of about 0°. Bob (Alice'sfriend) holds her smartphone and views its display that shows him whereto position the smartphone with respect to Alice's face. Bob moves aboutfour feet from Alice and holds her smartphone out in front of himself sothe speaker is pointed at her face. The smartphone determines it islocated three feet and five inches from Alice's face, and this positionis correct since it is within an acceptable range for localizing voicecalls. The smartphone also determines that it is located at an azimuthangle of +28° and at an elevation angle of 3° from Alice's face. Theseangles are within an acceptable range to position an SLP for voicecalls. Upon arriving at this position, the smartphone instructs byannouncing: “Taking picture in three, two, one.” One second later aspeaker in the smartphone generates a distinct tone that is specific forgenerating audio impulse responses that can be used to convolve sound.Microphones in Alice's ears capture this tone, and the smartphonegenerates BRIRs for the current location and stores them as SLP 2. Thesmartphone instructs Bob to move to the left side of Alice so it cangenerate another tone and capture Alice's BRIRs for SLP 3. Thereafter,when Alice receives a voice call, her smartphone convolves the soundaccording to the measured impulse responses so the voice of the callerlocalizes to one of SLP 2 or SLP 3.

Alternatively Alice's smartphone can work in cooperation with Bob'ssmartphone to direct Bob to hold it at the designated position andgenerate the audio impulse from Bob's smartphone or from Bob himself.

In another example embodiment, the HPED does not produce the audioimpulse itself at a designated instant but instead instructs the user tocause the audio impulse to be emitted. For example, the HPED enters amode of readiness to capture the audio impulse and the HPED informs theuser to cause the audio impulse within twenty seconds. The user cancause a sound that the HPED will recognize by prearrangement as thetarget impulse. The target impulse is the designated sound that the HPEDwill analyze in order to create the BRIRs and HRTFs. As an alternativeexample, the HPED informs the user that it is in the ready state for thenext twenty seconds and directs the user to cause the audio impulse. TheHPED then analyzes the impulses and selects the target impulse that issuitable to use in the creation of a set of BRIRs and/or HRTFs. Asanother example, the HPED can be set in a ready mode to capture targetimpulses that occur in a prearranged zone such as an approximate azimuthangle or with an approximate ITD. Further, the HPED can be designated inits ready mode to analyze the first impulse following a certain spokentrigger word or sound, or following a voice signal that exceeds acertain volume, such as double the average volume of the backgroundnoise. The examples above illustrate that an example embodiment cancapture and record the sounds at a location for various lengths of timeand then select the optimal impulse to use in creating the BRIRs andHRTFs. The HPED can also be instructed to disregard impulses accordingto a set of criteria, and/or to consider for analysis impulses thatmatch a set of criteria and disregard ones that do not match thecriteria.

In another example embodiment, the HPED can use the set(s) of criteriain order to select the target impulse, however the HPED is not commandedto enter a ready state. Instead, an application executing on the HPEDcontinually or periodically captures, records, and erases sound from theenvironment. Memory retains a certain number of seconds of the recordedsound prior to any moment in a dynamic cache. The applicationcontinuously monitors the cache and analyzes the recorded sound toidentify one or more keywords or key sounds. The keywords or key soundscan be the target impulse(s), or they can indicate that the targetimpulse is following or has preceded the keyword or key sound. Upon theidentification of a target impulse, the application stores the impulseto memory and thereafter processes saved impulses into BRIRs and/orHRTFs and enters them in the SLP database. Consider an example in whichthe following sounds are designated as target impulses: the sound of arubber band snapping against a book, the sound of the soles of two shoesclapped together, the sound of an isolated hand clap, the sound of atongue click, the sound of a wine glass clink, the sound of a car keyrap on a glass pane.

Consider an example in which Alice and two friends are seated in arestaurant. Alice wishes to create a designated SLP for each of them.Alice says, “I have a new binaural phone. Please clap yourselves in.”Her phone is in the ready state to capture predefined target impulses,she is wearing her microphones at her ears, and keeps her head facingforward. Bob then claps his hands once and says, “I'm Bob.” Yoko says,“This is Yoko,” and claps. During this period the application identifiestwo separate target impulses and saves each one to memory for subsequentprocessing into SLPs. The SLPs are added to the database and associatedwith Bob and Yoko in her smartphone's contact list. Thereafter duringphone calls with Bob and/or Yoko she can localize each to their relativeposition to her when they were at the restaurant.

When an impulse response is captured in accordance with an exampleembodiment, a SLP is calculated for the impulse response and itscorresponding location, such as the (r, θ, ϕ) location where the HPEDwas when it generated the sound. Each SLP has unique characteristicssince each SLP can be captured at a different location (r, θ, ϕ),captured with a different head orientation of the listener, capturedwith a different sound, captured with a different speaker generating thesound, captured under different environmental conditions, etc. If theuser has one or two SLPs, then these SLPs and their characteristics canbe readily remembered or managed. As the number of SLPs increases,however, it becomes more difficult for the user to manage the SLPs anddetermine information such as which SLP corresponds to which location,what SLPs the user has, what areas near the user do not have SLPs, whatRIRs or HRIRs are associated with which SLPs, how accurate a particularSLP is, which SLPs a user prefers for use with particular headphones,which SLPs a user prefers for speaking with particular people, whichSLPs a user prefers at particular times of day or under particularcircumstances, where sound for each SLP actually localizes to the user,etc. Example embodiments assist in solving these and other technicalproblems.

FIG. 5 is a method to designate one or more SLPs to a zone or areaaround a head of a person. This method assists the person in managinghis or her SLPs.

Block 500 states divide an area around a head of a person into aplurality of three-dimensional (3D) zones.

Areas around the head of the person are divided into a plurality ofdifferent zones. The zones can have similar, same, or different sizesand shapes that include regular or irregular 3D shapes including, butnot limited to, one or more of a sphere, a cube, a cuboid, a cone(including truncated cones), arc, a cylinder, a pyramid (including asquare-based pyramid), a tetrahedron, a triangular prism, polyhedrons,uniform 3D shapes, hemisphere, a partial sphere, a portion or slice of asphere, non-uniform or irregular 3D shapes, and other shapes.

Consider an example in which the head of the person is centered at anorigin (0, 0, 0) of an X-Y-Z coordinate system or spherical coordinatesystem. An imaginary sphere of radius (r) encircles the head with thehead at the origin. This sphere is further divided into areas, pieces,or zones. For example, the sphere is cut or divided into a plurality ofhorizontal cross sections, vertical cross-sections, cones, or other 3Dshapes. Each zone represents a location where one or more SLPs can bederived or exist.

The following illustrates an example to define an area above a head ofthe person whose head is centered at the imaginary sphere. Consider anexample in which an imaginary horizontal plane cuts through this sphereabove the head of the person such that a spherical cap defines an areaabove this horizontal plane. This spherical cap can be further dividedinto zones by cutting the spherical cap with imaginary horizontal,vertical, or angled planes. One or more SLPs can be designated into eachzone. Alternatively, a zone can be designated with no SLPs.

Block 510 makes a determination as to whether the HPED is located in oneof the zones.

For example, when the head of the person is positioned at an origin (0,0, 0) of the coordinate system, then the location of the HPED can becalculated with respect to this position. For instance, its location (r,θ, ϕ) or (x, y, z) with respect to the face of the person (i.e., theorigin) can be calculated as discussed herein. The location is thencompared with the coordinates of the zones to determine into which zonethe HPED currently resides.

The HPED can actually have multiple location designations. Onedesignation is its location in space relative to the head of the person,such as designating the location of the HPED at (r, θ, ϕ) or (x, y, z).Another designation is its GPS or physical location, such as 22° 33′ and114° 14′. Another designation is a name of a physical location, such asan address, name of a building, name of a room, etc. Another designationis a name or designation of a zone or area around the head of theperson, such as an area around the head of the person having ten zonesand the HPED being located in Zone 3.

An example embodiment compares a current location of the HPED withrespect to the locations of the zones around the head of the person.This comparison reveals in which zone the HPED is currently located.

A determination as to whether the HPED is located in a zone and anidentification of that zone depends on a number of factors, such as thenumber of zones, the size of the zones, the shape of the zones, thedistance between the HPED and the person, etc.

Consider an example in which a head of the person is a center of spherethat is divided into multiple segments or zones. One of these zones(designated as Zone A) exists as a top portion of this sphere formedfrom a horizontal plane that dissects the sphere above the head of theperson (previously provided as an example of a spherical cap in thediscussion of block 500). If the sphere has a radius (R), and Zone A hasa height (h) and its own base radius (r), then the volume of Zone A(i.e., the spherical cap) is given by the following equation:V(Zone A)=⅙·π·h·(3r ² +h ²).Using the Pythagorean theorem (A²+B²=C²), results in the followingequation:(R−h)² +r ² =R ².Solving for the base radius (r), yields:r=(h(2R−h))^(1/2).

The angle (α) between the normal to the sphere at the bottom of thespherical cap and the base plane can be calculated with the followingequation:R−h=R sin α,α=sin⁻¹((R−h)/R)).Further, a geometric centroid (z) of the spherical cap (i.e., Zone A)occurs per the following equation:z=(3(2R−h)²)/(4(3R−h)).

These equations, along with the location of the HPED at (r, θ, ϕ) or (x,y, z), determine whether the HPED is located in the spherical cap, ZoneA.

Of course, these equations represent an example of how geometry andcoordinates can be used to determine whether the HPED is within aparticular zone around the user. Other equations and computations can beused and depend on the size, shape, and locations of the zones and HPED.Furthermore, other methods can be used to determine in which the zonethe HPED is located.

If the answer to this determination is “no” then flow proceeds to block520 that states take an action.

An action occurs when the HPED is not within a zone. For example, theHPED may be too far away from the person or may be too close to theperson. As another example, the HPED may be located in an area that doesnot include a zone, such as being under the person or in a pocket of theperson. Further, a zone may already have a sufficient number ofeffective SLPs.

Example actions include, but are not limited to, providing the user orother person with a sound warning (e.g., playing a sound warning fromthe HPED), providing the user or other person with a visual warning(e.g., displaying a visual indication on the HPED, such as a text orlight or an image), displaying a visual indication of the zone or zonesso the user can move to or navigate to the zone and correct location,providing verbal instructions that indicate where the user shouldphysically move so the user and/or HPED is within a particular zone,preventing the HPED from generating the sound to capture the impulseresponse (e.g., the HPED will not automatically generate the sound),allowing the HPED to generate the sound (e.g., the HPED generates thesound but notes a warning or designation with the facial orientation),instructing the user or other person to change the facial orientation orthe orientation of the HPED (e.g., providing a written message orannouncement from a voice to move or rotate the HPED and/or change ahead orientation of the person), capturing multiple impulse responsesfrom the same location or other locations proximate to the location(e.g., capturing more impulse responses when the listener and/or HPEDdoes not have a specified or correct orientation), processing theimpulse responses to compensate for the facial orientation, or takinganother action.

If the answer to this determination is “yes” then flow proceeds to block530 that states generate a sound from the HPED to capture impulseresponses for the person.

The HPED generates the sound to capture the impulse responses at themicrophones located in the ear of the listener.

Block 540 states designate the zone as including a sound localizationpoint (SLP) for the person to localize binaural sound.

A record of information is kept with respect to each zone. Thisinformation includes, but is not limited to, one or more of a locationof a zone, a size and shape of a zone, a creation date and time of azone, a number of SLPs in the zone, a location of each SLP in the zone(e.g., in what part of the zone is the SLP located), a number of timesand duration of time a zone or SLP in the zone is used to localizesound, which sounds and sound types localize to which SLPs, aneffectiveness or accuracy of a user to localize sound to the SLP in azone, voices or people or contacts designated to a SLP, names of eachzone and each SLP, and other information discussed herein.

Block 550 states store the information for the zones and the SLPs.

This information includes information discussed herein with respect tothe zones and the SLPs.

The zones and SLPs can also be mapped to provide a user with a 2D or 3Dvisual indication of the zones and a location of the SLPs in the zones.For example, the HPED displays an image of the zones and where the SLPsare located in each zone. The image can also include other information,such as names, types, color descriptors, and other portions of theinformation.

Impulse responses can be particular to a location or type of locationwhen the impulse responses include noise reverberations, such as noisereverberations caused from a size of the location, a shape of thelocation, objects at the location, environmental conditions at thelocation, etc. These impulse responses occur in the form of RIRs thatare included in the BRIRs captured at the microphones of the ears of thelistener when the HPED generates the sound. RIRs can also be added to orremoved from the sound after the impulse responses are recorded, such asadding a high-ceiling effect so the sounds appear to originate in acathedral.

Problems can exist when a user has many different stored impulseresponses for different occasions, different locations, differentpurposes, etc. and it can be difficult to manage these various impulseresponse sets or transfer functions associated with the impulseresponses. For instance, a user could have a series of SLPs for voicecalls, and each of these SLPs can have a different BRIR. Some SLPsprovide the listener with a sound effect so that a voice of the speakerappears to originate at the beach, at an office, at a home, or atanother location. Further, the user could have designated or capturedsome SLPs at his house and captured others at his office. Each SLP canhave a set of rules to determine when it should be activated to localizea sound.

Example embodiments, including those discussed in FIGS. 6 and 7, addressthese technical problems and others discussed herein.

FIG. 6 is a method to map a SLP to a location.

Block 600 states generate a sound from a HPED to capture impulseresponses of a person wearing microphones when the HPED is proximate tobut away from the person.

The HPED generates a sound or causes another electronic device togenerate a sound that is used to capture impulse responses atmicrophones located in, at, or near the ears of the person.

Block 610 states store a location of the person and/or the HPED when theHPED generates the sound.

A location of the HPED and/or person is stored and can be retrieved,processed, transmitted, etc. The information stored is not limited tolocation, but also includes a facial or head orientation of the person,coordinate information regarding the person and/or HPED (including (r,θ, ϕ) or (x, y, z) discussed herein), an address of the person and/orHPED when the impulse response was generated, a description or anidentification of the location (such as labeling the location accordingto room type, like “bedroom” or “office”), and other informationdiscussed herein.

Block 620 states generate a SLP at the location of the HPED when theHPED generated the sound for the impulse responses.

When sound is subsequently convolved with the impulse responses for thislocation, the sound will appear to originate relative to the listener atthe location of the HPED at the instant in time when the HPED generatedthe sound. Example embodiments set or establish a SLP for this location.

Further, a SLP can be provided with a descriptive name so a user canrecognize the SLP and/or its location. For instance, a user captures animpulse response in her bedroom when her HPED is four feet from herface. The location of the HPED represents where she wants to localizeher husband's voice when he calls her. A SLP designates to this locationand is stored as “Telephony Husband” so she can distinguish this SLPfrom other SLPs that are designated to her.

Block 630 states store the location and other information.

The information is stored in the HPED and/or stored in another location,such as stored on a server, another electronic device, a database,memory, a cloud, etc.

Block 640 states provide the SLP to a map for subsequent retrieval.

In addition to storing the information, it can be provided to the personin a visual and/or audio context. An example embodiment maps the SLP andother information into a 2D or 3D map so the person can easily andquickly see the SLP and relevant information associated with it. The SLPand accompanying information can be retrieved and viewed on or through adisplay, such as being viewed on an electronic device (such as acomputer or television), a HPED, electronic glasses, a head-up displayor other display adapted for virtual reality (VR) or augmented reality(AR), or other type of wearable electronic device.

Consider an example in which Alice wears earphones with microphones anduses her HPED to capture numerous BRIRs in different rooms while in herhouse. A SLP designates for each BRIR. An example embodiment builds orretrieves a 3D interactive map of her house and places each SLP at itscorresponding location in the map. Alice displays the map on her HPEDand sees where each SLP is located. Further, the HPED knows the locationof each SLP in order to select or recommend a SLP for Alice, such as anintelligent personal assistant recommending or selecting a SLP forAlice.

FIG. 7 is a method to select a SLP for a person and convolve sound tolocalize at the SLP.

Block 700 states determine a location of a person.

Example methods to locate a person include, but are not limited to,tracking a person and/or HPED with GPS, tracking a smartphone with itsmobile phone number, tracking a HPED via a wireless router or wirelessnetwork connection to which the HPED communicates for Internet access,tracking a person and/or HPED with a tag or barcode, tracking a personand/or HPED with a radio frequency identification (RFID) tag and reader,tracking a location of a person with a camera (such as a camera inconjunction with facial recognition), and tracking a location of aperson with a sensor. Alternatively, a person can provide his or herlocation (such as speaking a location to an intelligent personalassistant that executes on a HPED).

Consider an example in which a smartphone executes an application thattracks and shares its current location in real-time with otherapplications, electronic devices, and/or example embodiments discussedherein.

Block 710 makes a determination as to whether one or more SLPs exist forthe location.

SLPs can be stored or associated with locations, including zones, areas,places, rooms, etc. When a person goes to or near a location, then theSLPs associated with this location are retrieved. For example, a HPED ofa person compares its current location with the locations of SLPs storedfor the person to determine whether one or more SLPs exist for thelocation.

The determination as to whether a SLP exists for a particular locationcan be based on one or more factors. These factors can determine how orwhich SLPs are selected.

For example, one factor is proximity of the person and/or HPED to theSLP or location where the impulse responses associated with the SLP weregenerated. A SLP can be selected based on its proximity to the personand/or HPED. For instance a SLP closest to the person and/or HPED isselected.

Another factor is the RIR associated with the SLP. For example, aclosest SLP may not be appropriate if this SLP has an RIR that is notassociated with the current location of the person. Consider an examplein which Alice has many SLPs throughout her house. Each SLP includesRIRs for the particular room in which the SLP is located. SLPs in thebathroom are convolved with bathroom RIRs; SLPs in the bedroom areconvolved with bedroom RIRs; etc. When Alice receives a call, the voiceof the caller is convolved with a RIR corresponding to the location ofAlice. While standing in the hallway, Alice receives a call from Bob onher smartphone. The closest SLP is a bathroom BRIR that is located a fewfeet from Alice. Since Alice is not in the bathroom, her smartphoneselects a bedroom BRIR since the HPED senses her walking direction andpredicts she will enter this room shortly and not the bathroom.

Another factor is historic usage or personal preferences. When theperson was previously at this location, he or she localized sound with aparticular SLP and BRIR, and this SLP and BRIR are recommended for thislocation based on the past selection. For example, a user has a favoriteSLP to use for voice calls, or has a specific SLP used for calls with aparticular friend regardless of their location at the time of a call.

If the answer to this determination is “no” then flow proceeds to block720 that states take an action.

An action occurs when a SLP or impulse response does not exist for thecurrent location of the person. For example, the person enters a room orlocation for the first time, and no RIRs or BRIRs exist for thislocation.

Example actions include, but are not limited to, choosing a genericimpulse response in order to convolve the sound (e.g., choosing a BRIRtaken from or associated with another physical location); choosing a RIRor BRIR not particular to the location but associated with the location(e.g., when the person is in a car for which no RIR exists, thenchoosing a RIR from another car); instructing the user to capture a BRIRfor this location; playing a particular ringtone that signifies to theuser that a SLP or impulse response is not available for the currentlocation; selecting to localize the sound at a predetermined locationwith no RIR information (e.g., localize the sound with individualizedHRTFs of the user that do not include RIRs); providing the user or otherperson with a sound warning, providing the user or other person with avisual warning, denying the HPED from localizing sound (e.g., providingthe sound in stereo or mono to the person instead of providing binauralsound that localizes to an external location); instructing the user orother person to move to another location where a SLP or impulse responsewas previously captured for the person; or taking another action (suchas an action discussed herein).

If the answer to this determination is “yes” then flow proceeds to block740 that states select a SLP to localize the sound to the person.

An electronic device chooses one or more available SLPs and theirassociated impulse responses or transfer functions to convolve sound sothe sound localizes to the selected SLP.

Block 750 states convolve the sound with the set of impulse responses ortransfer functions associated with the selected SLP so the soundlocalizes to the SLP that is proximate but away from the person. Soundis convolved so it localizes to the person at the SLP.

Voice telephony is more realistic when the voices are localized toinclude RIRs for the current location of the listener. One problem isthat the electronic device of the listener may not have RIRs for hiscurrent location and hence cannot convolve sounds with impulse responsesfrom the location.

FIG. 8 and other example embodiments address this problem and others.

FIG. 8 is a method to retrieve RIRs and convolve sound based on alocation of a person.

Block 800 states determine a location of a person. The location of theperson can be determined as described in connection with block 700 orother blocks discussed herein.

Block 810 makes a determination as to whether a RIR exists for thelocation.

RIRs can be stored and associated with locations. When a person goes toor near a location, then the RIRs associated with this location orlocation type are retrieved. For example, a HPED of a person comparesits current location with the locations of stored RIRs available locallyand online and determines whether one or more RIRs exist for thelocation or are suitable for the location.

In one example embodiment, the HPED or other electronic device of theperson captures the RIRs for the location. For example, while the personis at the location, a HPED of the person generates a sound, andelectronic microphones capture impulse responses for the sound. Inanother example embodiment, the HPED or other electronic deviceretrieves RIRs for the location. For instance, RIRs are stored in adatabase or memory for various locations around the world, and theseRIRs are available for retrieval. These RIRs can be actual ones capturedat the location or computer generated or estimated RIRs for thelocation. As yet another example, the HPED or electronic deviceretrieves RIRs for a similar location. For instance, if the location isa church but no RIRs exist for this particular church, then RIRs foranother church are retrieved. Physical attributes of the location (suchas size, shape, and other physical qualities) can be used to moreclosely match RIRs from other locations.

In example embodiments, reverberation can be physically measured ordigitally simulated. For example, to apply a reverberation effect, anincoming audio signal is convolved with an impulse response. Convolutionmultiplies the incoming audio signal with samples in the impulseresponse file. Various impulse responses for specific locations (rangingfrom small rooms to large areas) can be retrieved from memory and thenused in convolution reverb applications to provide an audio signal withacoustic characteristics that are particular to the specific location.

Consider an example in which a transfer function or frequency responsefor an area (such as a room or other location) is measured with a soundthat covers the frequency spectrum. For instance, a white noise excitesthe area, and the noise is recorded at locations near the source andanother location in space. Coefficients of an impulse response aregenerated as an inverse Fourier Transform.

Consider an example of convolution reverb in which one or moremicrophones are placed in a room and a brief pulse sound is generated.The microphones capture both the original sound and the response orreverberations from the room to generate RIRs for the room.

If the answer to this determination is “no” then flow proceeds to block820 that states take an action.

An action occurs when an RIR or impulse response does not exist for thecurrent location of the person. For example, the person enters a room orlocation for the first time, and no RIRs or BRIRs exist for thislocation.

Example actions include those discussed in connection with block 730and/or taking another action (such as an action discussed herein).

If the answer to this determination is “yes” then flow proceeds to block830 that states convolve sound with the RIR.

Block 840 states provide the convolved sound to the person.

For example, an electronic device convolves the sound and provides it tothe person through speakers, such as speakers in earphones, wearableelectronic device, or loud speakers.

Consider several examples in which Alice prefers to localize voices onphone calls with RIRs captured from the physical location where she istalking to increase realism. This preference is set on her smartphone.

Alice receives a VoIP call from Bob while she is at her grandmother'shouse. Her smartphone determines that Alice has not previously receiveda call at this location and hence is unable to retrieve either a RIR orBRIR for her current location. In response to this determination, thesmartphone rings with a distinctive tone, and Alice recognizes this toneand its implication that no RIRs or BRIRs are available for herlocation. This distinctive tone is actually the sound used to captureimpulse responses. While her smartphone is ringing and generating thisdistinctive tone, Alice holds the smartphone in her hand with her armstretched out away from her face. Microphones in her earphones recordthe tones, and her smartphone immediately generates BRIRs for Alice.When the smartphone captures sufficient impulse response from adesignated location, it stops generating the tones, answers the call,and convolves the incoming voice with the BRIRs that it just obtainedwhile Alice was answering the phone call.

Consider the example above in which Alice is at her grandmother's housewhen she receives a voice call but her smartphone does not have RIRs orBRIRs for her location. Alice answers the call and talks with a headsetthat includes microphones and speakers. A voice of the caller localizesto a SLP that is proximate to Alice with her HRTFs, but the voice is notconvolved with RIRs because they do not exist for her current location.During the call, Alice asks Bob to generate a RIR reference impulse. Bobwalks several feet away from Alice and activates his phone to generate adistinctive tone. The microphones in Alice's ears recognize the tone asone to generate impulse responses for her current location. Based onthese impulse responses, her smartphone generates BRIRs particular toAlice and her location and then determines the RIRs for the room usingthe new BRIR and her known HRTF. Alice continues the conversation withconvolution now including the room's RIR, and without changing her SLP.

Consider the example above in which Alice is at her grandmother's housewhen she receives a call but her smartphone does not have RIRs or BRIRsfor her location. Her smartphone determines that Alice has notpreviously received a call at this location and rings with a distinctivetone. This tone, however, is not the tone to capture the impulseresponses for Alice and the room. Instead, the tone alerts Alice thatsuch impulse responses are missing or that she is missing individualizedHRIRs for her location. Alice moves the smartphone to a locationproximate to her face to where she would like to localize the voice ofthe caller. The smartphone continues to ring, but tracks its location.When the smartphone moves to the correct location (e.g., to a particularzone prearranged by Alice or to a location to measure a far-field HRTF),or when Alice indicates the location is correct, the smartphonegenerates a specific tone designed to capture audio impulse responses,such as generating a frequency-swept sine wave or other sound. Based onthese impulse responses, her smartphone generates BRIRs particular toAlice and her location. Alice answers the call, and her smartphoneconvolves the caller's voice with the newly captured BRIRs and localizesit to a SLP that is away from but proximate to Alice.

FIGS. 9A-9D show a person 900 standing next to a table 902 and capturingaudio impulse responses with microphones 910A and 910B in his left andright ears. The person 900 holds a hand held portable electronic device(HPED) 920 in his hand 930 with his left arm 940 straight and extendedaway from his body.

When the left hand 930 is located away from the face of the person 900with the arm in an outstretched position, the HPED 920 (such as asmartphone) generates a specific sound or tone to generate audio impulseresponses at the microphones 910A and 910B. The HPED 920 or anotherelectronic device processes the impulse response to generate HRIRs,BRIRs, RIRs, and/or HRTFs that are used to convolve sound to the person.

When sound is subsequently convolved with the impulse responses ortransfer functions, the sound localizes for the person 900 to thelocation where the HPED 920 was when it generated the sound or tone.These locations are stored as sound localization points or SLPs.

FIGS. 9A-9D show a plurality of SLPs 950 formed around the person 900.Each SLP represents a location where the person generated a sound ortone with the HPED 920 and captured the impulse response with themicrophones 910A and 910B located in his ears. By way of example, thefigures show the person 900 activating the HPED 920 to generate a soundor tone at SLP 952. The HPED 920 or another electronic device (such as acloud server) stores the locations of the SLPs and other informationassociated with them.

As shown in FIGS. 9A-9D, the SLPs 950 form a pattern (such as a partialsphere) around the head or body of the person 900. A first set of SLPs954 form a partial sphere around a left side of the person, and a secondset of SLPs 956 form a partial sphere around a right side of the person.SLPs 954 were generated when the HPED 920 was extended outwardly in theleft hand 930 of the person 900, and SLPs 956 were generated when theHPED 920 was extended outwardly in the right hand of the person 900. Alength of each arm serves as a radius for the partial spheres.

FIGS. 9A-9D show the SLPs formed around the person in a structuredpattern. The SLPs, however, can occur at various locations, such asarbitrary locations where the person places the HPED and generates thetone to capture the impulse responses with the microphones.

Example embodiments enable the person to select locations for where togenerate the SLPs. The person can select not only where to position aSLP but also how many SLPs to generate and/or store. For example, oneperson may want to generate one or two SLPs for localizing sound, whileanother person may want to generate hundreds of SLPs for localizingsound from one or more distances.

Example embodiments are not limited to capturing impulse responses andgenerating SLPs while a HPED is being held in a hand of a person. Peoplehave significant flexibility in deciding where to place a SLP such asdeciding distances and angles of a location, and one or more of (r, θ,ϕ). This flexibility further includes allowing people to place SLPs at,on, or near objects, such as placing their HPED on an object andgenerating the sound from this location. FIGS. 10-12 illustrate someadditional examples of these flexibilities.

FIGS. 10A-10D show a person 1000 standing and capturing audio impulseresponses with microphones 1010A and 1010B in his left and right ears.The person 1000 holds a pole, rod, or selfie-stick 1020 that attaches orconnects to a HPED 1030. A plurality of SLPs 1040 encircles or surroundsthe person 1000.

When the HPED 1030 is connected to one end of the pole 1020, the person1000 can position the HPED at a much farther distance from his face thanif he were holding the HPED in his hand and extending his arm. In thismanner, the person can generate and capture far-field HRTFs since thecombined length of his arm and the selfie-stick is greater than about1.0 meter. Depending on the length of a person's arm and a length of thepole, the person can generate and capture impulse responses from 1.0-2.0meters or more from his ears. For example, a person could generatesounds with a smartphone positioned about three feet to about eight feetfrom the face of the person. The pole thus ensures that the persongenerates and captures far-field audio impulse responses and providesmore flexibility to generate SLPs in a number of locations.

FIGS. 10A-10B show an example configuration of SLPs 1040 that werecaptured with left and right arms of a person 1000. A right hemisphereof SLPs 1050 forms on a right side of the person, and a left hemisphereof SLPs 1052 forms on a left side of the person.

As noted, a spherical configuration of SLPs is an example shape sincethe person can place SLPs at a variety of different locations accordingto his or her desires. SLPs are not restricted or confined to aparticular distance or configuration around the person. Instead,locations and numbers of the SLPs are user-selected. This is contrary totraditional systems (such as an anechoic chamber) in which the numberand location of the speakers (which coincide to the SLPs) were fixed andnot user-selected.

FIG. 11 shows a person 1100 sitting at a table 1102 and capturing audioimpulse responses with microphones 1110A and 1110B in his left and rightears. While a HPED 1120 sits on the table 1102, the HPED generates atone or sound that the microphones capture as the impulse responses. Thelocation of the HPED when it generates the sound represents the SLP towhere subsequent sound is convolved and localized for the person.

FIG. 11 shows two SLPs 1130 and 1132 that were previously generated whenthe HPED was at the respective locations. For instance, when the HPEDwas placed next to the lamp 1140, the HPED activated the sound togenerate the impulse response, and the SLP 1132 was subsequently createdfor that location. Likewise, when the HPED as placed next to thecomputer 1150, the HPED activated the sound to generate the impulseresponse, and the SLP 1130 was created for that location. The HPED iscurrently resting next to some books 1160 and is ready to generateanother SLP for the person.

FIG. 11 shows that a person is not required to hold the HPED whengenerating the tone and capturing the impulse responses. Instead, theHPED is set or placed at a location, and it generates the tone from thislocation. This embodiment provides the person with much flexibility indetermining where to generate a SLP. The person in FIG. 11, for example,can generate SLPs on his table and position these SLPs at locationsconvenient or desirable to him. For example, the person 1100 candesignate the SLP 1132 as a “virtual right speaker” and a SLP createdfrom an impulse response at the HPED 1120 as a “virtual left speaker.”FIG. 12 shows a person 1200 sitting in an automobile 1210 and capturingaudio impulse responses with microphones 1220A and 1220B in his left andright ears while holding a HPED 1230 in his right hand 1240. The lefthand 1250 remains on the steering wheel 1260 of the automobile as theperson continues to drive and look straight ahead toward the road.

The HPED 1230 generates a specific sound or tone used to generate audioimpulse responses that can be used to create localized sound at a SLP.In this instance, the SLP is located to a right side of the person andat a passenger seat 1270 of the automobile. Subsequent sounds (includingvoices) can be localized to this SLP at the passenger seat 1270. Forexample, the person can communicate with an intelligent user agent orautopilot whose voice localizes to the SLP at the passenger seat orcommunicate with another person during a phone call with the otherperson's voice localizing to the passenger seat 1270.

Example embodiments enable users to generate SLPs at locations specifiedor desired by the users. Furthermore, these SLPs can be close to theperson (such as near-field locations) or farther from the person (suchas far-field locations). Further yet, users can generate a SLP andimmediately begin to localize sound to this location. For example, oncethe microphones detect the impulse responses, the HPED processes them toderive transfer functions. The HPED then convolves sound input signalswith the transfer functions so the sound localizes to the location ororigin of the sound (i.e., where the HPED or other sound source wasphysically located when it generated the sound for the impulseresponses).

FIG. 13 shows a person 1300 sitting in a chair 1310 at a table 1320 andbeing surrounded by a plurality of SLPs 1330 illustrated as squareblocks that are located away from but proximate to his face. The person1300 holds a HPED 1340 in his left hand 1350 and points the HPED at oneof the SLPs, shown as SLP 1360. Pointing the HPED at the SLP in effectselects this SLP for sound localization for a source he designates.Thereafter, sounds will localize to this selected SLP, such as soundsheard through a wearable electronic device 1370. The wearable electronicdevice also displays the SLPs 1330 to the person.

SLP 1360 is darkened when compared to the other SLPs 1330 to visuallydistinguish it from the other SLPs as being the one selected for soundlocalization. By way of example, SLP 1360 can be distinguished withshading, light, color, indicia, symbols, text, or other visuallyrecognizable forms to signify its selection and to distinguish it fromnon-selected SLPs. Further, the SLPs 1330 are viewable on or through adisplay, such as a display of the HPED 1340, wearable electronicglasses, or another electronic device.

Consider an example of a phone call that originates from Bob to Alice,both of whom subscribe to a single channel monophonic mobile phonecellular network. They both have smartphones with stereo capability toplay music, and they both have stereo earphones to listen to the musicand to take telephone calls. Bob originates the call to Alice with thesmartphone's stock phone application and waits while he hears the ringindicator. Alice is driving her car wearing headphones and is listeningto her phone playing music when she hears a ringtone. The ringtoneindicates the she does not have an SLP configured for her currentlocation on the road. She also has not yet taken a call using a SLP withher new phone application that supports binaural speech convolution. Sheis already wearing her headphones with microphones so she takes thisopportunity to create a SLP suitable to use in the car so she can enjoya more natural phone conversation with the perception of Bob's voiceexternalized. On the display of her phone there is an “answer phone”button/option and a button/option that says, “answer at new SLP.” Aliceselects the latter option to answer at a new SLP. Her phone indicatesthat it will generate an SLP when the phone is steady at arm's length.Bob is then connected and they exchange greetings. Soon Alice tells Bob,“Hold on for a moment, I'm in a car and I'd like to externalize you . .. ” She extends her right arm toward the passenger seat while keepingher face safely toward the road. The phone's binaural callingapplication monitors the image received by the phone's camera. When theapplication detects Alice's facial profile in the center of the image,the application uses the image to calculate the phone's locationrelative to the face of Alice and determines the distance to her face tobe arm's length. The phone further uses its motion detector to determinethat it is steady and provides an indication (e.g., vibratory or audio)to Alice that it is ready to create the SLP.

Alice's phone then emits a short tone repeatedly and captures the audioimpulse responses with the microphones in Alice's ears as she faces theroad. The application then creates a SLP and a new SLP record in memoryand stores a timestamp, the new transfer function, the captured photo ofAlice's profile, the GPS location at the time of the tone capture, andthe position of the phone at the time of tone. The phone also recognizesthe brand and model of Alice's headphones so it creates an additionalcommon transfer function (CTF) and stores the CTF as well as the modelof the headphones with the SLP record.

The sound from Bob's call is optimized for speech by his phone'shardware and signal processing that has removed noise and non-speechfrom the sound. Alice's application convolves the sound to localize atthe SLP while the call conversation continues.

When Alice receives subsequent voice calls, her intelligent user agent(IUA) selects the transfer function that pre-calculates allowance forher headphones if she is wearing them at that time.

Bob is not using a phone application that can convolve Alice'smonophonic voice, so he hears her voice internalized (i.e., inside hishead). Later Alice uses the same phone to place a call to Bob from herhome while she sits at her kitchen table and wears headphones. Her phonechecks her GPS location and finds a SLP record that corresponds to Boband her location at the kitchen table. A binaural phone applicationexecuting in her phone selects this SLP to convolve the call with Bob.When Alice places the call to Bob, she hears the ring localize to theSLP that is away from but proximate to her. This external localizationnotifies her that this location will be used as the SLP for Bob's voicewhen he answers. When Bob answers, Alice hears him speak from across thekitchen table at the SLP.

After their telephone conversation finishes, Bob downloads a callconvolving application to his smartphone. Later, Alice calls Bob whilehe is at a café. Bob has not prepared a SLP and is unsure how to createone. During the phone call, he asks Alice to help him establish a SLP.Alice says, “I'm here at my kitchen table. Since you are at the café, Iwill send you my SLP. It might not fit great but it'll probably work.”Alice tells Bob she will reveal her SLP location so he can see it. Sheallows permission for the other party, Bob, to see the relativepositions of herself and where she has positioned the SLP of Bob. Bobcan see an illustration on his phone of the relative positions of Aliceand the SLP of his voice. He sees that Alice is about three feet awayand a little to his right. Alice instructs him to select, “Enforcecongruence” and he does.

In order to make the call congruent, Bob's IUA searches his SLP databasefor an SLP that can convolve Alice's voice to a zone about three feetaway from him and a little to his right (which corresponds to Alice'sposition relative to him). Bob's SLP database, however, has no such SLPor available record, so his IUA requests a congruent SLP from Alice'scall convolving application. In response to this request, Alice's callconvolving application sends Bob's call convolving application thetransfer function she is using (without her headphone CTF responsemodification). Bob's call convolving application uses the transferfunction received from Alice, but swaps the left and right channels.This swapping happens in real-time when Bob selects “Enforce congruence”on his phone, and suddenly the voice of Alice moves from inside his toan external localization point a few feet in front of him slightly tothe right. Bob talks to Alice with her voice localized across from himat the café table.

Later during their phone call, Alice moves to her bedroom and reclineson the bed with her face toward the ceiling. She still perceives hisvoice about three feet in front of her between her face and the ceilingrendered in an acoustic environment of her kitchen. Both of these audioattributes are irritating to Alice, so she issues a command to her phoneto scan for an alternate SLP. An intelligent personal assistant (IPA) inher phone suggests several different SLPs from her database. Her IPAbriefly convolves Bob's voice to each of the SLPs so Alice can hear theeffects. Alice provides a verbal command to her IPA to select one of thealternate SLPs. The newly selected SLP suits her because as she lies onher back looking at the ceiling, she can hear Bob from beside her on thebed, facing her. This newly selected SLP also provides a cozy,attenuating, audio environment that gives her conversation with Bob amore intimate setting.

After the conversation Bob says goodnight and terminates the call. Alicehears a designated call termination sound that allows her to confirmthat Bob is disconnected and not just pausing. The call terminationsound begins at the SLP and transforms into a non-localized sound thatis internalized by Alice, reminiscent of how monaural calls terminatefrom inside the head.

Consider an example of a Voice-over Internet Protocol (VoIP) calloriginating from Bob to Alice over a Transmission ControlProtocol/Internet Protocol (TCP/IP) network. They both have smartphoneswith stereo capability to play music, and they both have stereoearphones to listen to music and to take telephone calls. Theirearphones include binaural microphones. Bob originates the call to Alicewith a Session Initiation Protocol (SIP) client that can transmit twofull-duplex channels, sending the input from microphones at theearphones he is wearing, and receiving the left and right channels ofthe other party at his left and right ears. He initiates the VoIP callto Alice by selecting her phone number from the directory displayed in aSIP client application. This application initiates the call by firstlogging into a designated telephony switch that supports SIP connectionsand also makes available a stereo or binaural codec.

Bob is at his house holding a birthday party for Alice with some of herfriends. Alice, however, cannot come to the party. Alice enjoys hearingthe binaural sound captured at Bob's ears and streamed to her throughthe stereo codec without alteration. As he walks around the room talkingwith her friends, Alice can localize the different voices at the party.Bob is her audio avatar during the call.

While the party is going on, Alice is in a noisy hotel lobby and Bob isnot interested in experiencing the localization of the sounds in herphysical environment. Instead, Bob prefers to hear her speech withoutother noise so he can speak with her. So he gives a command to hissmartphone that causes it to use a signal processing unit to dynamicallyseparate her sound signals and to remove all sound except the sound ofher speech. The smartphone also moves the sound of her speech to an SLPpositioned directly to his left, with the angle of projection of hervoice being directly forward, parallel to the direction of theprojection of his own voice. Sound localization with this orientationgives Bob a familiar spatial configuration in which Alice accompanieshim on his left, pacing him, and strolling with him around the room atthe party.

To accomplish this localization, Bob first creates a SLP to his left byissuing to his smartphone the voice command, “Move voice.” This commandtriggers the smartphone to go into tone capture mode to generate a newSLP. He would like to place the SLP one meter away, which is longer thanhis arm, so he asks a friend to position his phone one meter to hisleft. A sound convolving application that executes in his smartphonecreates a sound and proceeds to transform the impulse responses into atransfer function and generate a new SLP that corresponds to thelocation one meter to his left. A moment later the SLP is created andthe sound convolving application determines that the single voice beingreceived by the phone is the desired source to play at the new SLP.Suddenly, Bob hears Alice speaking at his side, facing the samedirection that he faces. He walks through the party with his invisiblefriend, the guest of honor, Alice, whom only he can hear.

FIG. 14 is a computer system or electronic system 1400 that includes ahandheld portable electronic device or HPED 1402, a computer orelectronic device (such as a server) 1404, electronic earphones 1406,and storage or memory 1408 in communication with each other over one ormore networks 1410.

The handheld portable electronic device 1402 includes one or morecomponents of computer readable medium (CRM) or memory 1420, a display1422, a processing unit 1424, one or more interfaces 1426 (such as anetwork interface, a graphical user interface, a natural language userinterface, a natural user interface, a reality user interface, a kineticuser interface, touchless user interface, an augmented reality userinterface, and/or an interface that combines reality and virtuality), acamera 1428, one or more sensors 1430 (such as micro-electro-mechanicalsystems sensor, a biometric sensor, an optical sensor, radio-frequencyidentification sensor, a global positioning satellite (GPS) sensor, asolid state compass, gyroscope, magnetometer, and/or an accelerometer),a sound localization system 1432 (such as a system that localizes sound,adjusts sound, predicts or extrapolates characteristics of sound,detects specific audio impulse responses, and/or executes one or moremethods discussed herein), an audio impulse response signal or soundgenerator 1434, a facial orientation system and/or facial recognitionsystem 1436, a head/eye tracker 1438, a location determiner 1440 (suchas hardware and/or software discussed herein to determine or tracklocation), microphones 1442, speakers 1444, and a battery or powersupply 1446.

The storage 1408 can include memory or databases that store one or moreof SLPs (including their locations and other information associated witha SLP including rich media such as sound files and images), userprofiles and/or user preferences (such as user preferences for SLPlocations and sound localization preferences), impulse responses andtransfer functions (such as HRTFs, HRIRs, BRIRs, and RIRs), and otherinformation discussed herein.

The network 1410 can include one or more of the Internet, a local areanetwork (LAN), a wide area network (WAN), a metropolitan area network(MAN), a personal area network (PAN), home area network (HAM), and otherpublic and/or private networks. Additionally, the electronic devices donot have to communicate with each other through a network. As oneexample, electronic devices can couple together via one or more wires,such as a direct wired-connection. As another example, electronicdevices can communicate directly through a wireless protocol, such asBluetooth, near field communication (NFC), or other wirelesscommunication protocol. One device can trigger another device usingsound waves.

The sensors 1430 can further include motion detectors (such as sensorsthat detect motion with one or more of infrared, optics, radio frequencyenergy, sound, vibration, and magnetism).

By way of example, a location determiner or location tracker includes,but is not limited to, a wireless electromagnetic motion tracker, asystem using active markers or passive markers, a markerless motioncapture system, video tracking (e.g. using a camera), a laser, aninertial motion capture system and/or inertial sensors, facial motioncapture, a radio frequency system, an infrared motion capture system, anoptical motion tracking system, an electronic tagging system, a GPStracking system, an object recognition system (such as using edgedetection), and other embodiments, such as hardware and/or softwarediscussed in connection with block 360 and other example embodiments.

The sound localization system 1432 performs various tasks with regard tomanaging, generating, retrieving, storing, and selecting SLPs. Thesetasks include generating audio impulse responses or transfer functionsfor a person, convolving sound per the impulse responses or transferfunctions, dividing an area around a head of a person into zones orareas, determining what SLPs are in a zone or area, mapping SLPlocations and information for subsequent retrieval and display,selecting SLPs when a user is at a determined location, and executingone or more other blocks discussed herein. The sound localization systemcan also include a sound convolving application that convolves soundaccording to one or more audio impulse responses and/or transferfunctions.

The audio impulse response signal generator or sound generator 1434generates a sound, tone, or signal that produces the audio impulseresponse captured at the microphones (such as the microphones being wornby a person). By way of example, such sounds include, but are notlimited to, a known spectrum stimulus sound, a frequency-swept sinewave, a click, a voice-like sound, a pulse, a maximum length sequence(MLS), a pseudo-random binary sequence, white or pink noise, a ping, acomplementary Golay code, a voice announcing a word or a phrase, oranother type of sound generated from a speaker to generate audio impulseresponses. The sounds can be generated by the sound generator and canalso be prepared sound files present on the HPED 1402 or received andupdated from time to time through the network 1410.

The facial orientation system and/or facial recognition system 1436recognizes faces of people, determines a facial orientation of a person,determines a distance from the HPED to a person, and performs otherfunctions discussed herein with regard to facial orientation and facialrecognition.

Electronic device 1404 includes one or more components of computerreadable medium (CRM) or memory 1460, a display 1462, a processing unit1464, one or more interfaces 1466, and a sound localization system 1472(such as a system that performs one or more functions discussed herein).

The electronic earphones 1406 include one or more of microphones 1480(such as left and right microphones that fit inside an ear of a person),speakers 1482 (such as a left and right speaker that are located in, at,or near an ear of a person), a battery or power supply 1484, and awireless transmitter/receiver 1486. The wireless transmitter/receivercan support audio streams discussed herein (for example, 4 simultaneousstreams, 2 channels out plus 2 channels in, at sample rates per by anexample embodiment) as well as other data.

By way of example, a computer and an electronic device include, but arenot limited to, handheld portable electronic devices (HPEDs), wearableelectronic glasses, watches, wearable electronic devices, portableelectronic devices, computing devices, electronic devices with cellularor mobile phone capabilities, digital cameras, desktop computers,servers, portable computers (such as tablet and notebook computers),electronic and computer game consoles, home entertainment systems,handheld audio playing devices (example, handheld devices fordownloading and playing music and videos), personal digital assistants(PDAs), combinations of these devices, devices with a processor orprocessing unit and a memory, and other portable and non-portableelectronic devices and systems.

The processor unit includes a processor (such as a central processingunit, CPU, microprocessor, application-specific integrated circuit(ASIC), etc.) for controlling the overall operation of memory (such asrandom access memory (RAM) for temporary data storage, read only memory(ROM) for permanent data storage, and firmware). The processing unitcommunicates with memory and performs operations and tasks thatimplement one or more blocks of the flow diagrams discussed herein. Thememory, for example, stores applications, data, programs, algorithms(including software to implement or assist in implementing exampleembodiments) and other data.

FIG. 15 is a computer system or electronic system 1500 that includes anelectronic device 1502, a server 1504, a wearable electronic device1508, earphones 1510, and speakers 1512 in communication with each otherover one or more networks 1514.

By way of example, electronic devices include, but are not limited to, acomputer, handheld portable electronic devices (HPEDs), wearableelectronic glasses, watches, wearable electronic devices, portableelectronic devices, computing devices, electronic devices with cellularor mobile phone capabilities, digital cameras, desktop computers,servers, portable computers (such as tablet and notebook computers),smartphones, electronic and computer game consoles, home entertainmentsystems, handheld audio playing devices (example, handheld devices fordownloading and playing music and videos), appliances (including homeappliances), personal digital assistants (PDAs), electronics andelectronic systems in automobiles (including automobile controlsystems), combinations of these devices, devices with a processor orprocessing unit and a memory, and other portable and non-portableelectronic devices and systems.

Electronic device 1502 includes one or more components of computerreadable medium (CRM) or memory 1515, one or more displays 1522, aprocessor or processing unit 1524, one or more interfaces 1526 (such asa network interface, a graphical user interface, a natural language userinterface, a natural user interface, a reality user interface, a kineticuser interface, touchless user interface, an augmented reality userinterface, and/or an interface that combines reality and VR), a camera1528, one or more sensors 1530 (such as micro-electro-mechanical systemssensor, an activity tracker, a pedometer, a piezoelectric sensor, abiometric sensor, an optical sensor, radio-frequency identificationsensor, a global positioning satellite (GPS) sensor, a solid statecompass, gyroscope, magnetometer, and/or an accelerometer), a locationor motion tracker 1532, one or more speakers 1534, impulse responses,transfer functions, and/or SLPs 1536, a sound localization system 1538(such as a system that executes one or more methods or blocks discussedherein), one or more microphones 1540, a predictor 1542, an intelligentuser agent (IUA) and/or intelligent personal assistant (IPA) 1544, soundhardware 1546, and a user profile builder and/or user profile 1548.

Server 1504 includes computer readable medium (CRM) or memory 1550, aprocessor or processing unit 1552, and an intelligent personal assistant1554.

By way of example, the intelligent personal assistant or intelligentuser agent is a software agent that performs tasks or services for aperson, such as organizing and maintaining information (emails, calendarevents, files, to-do items, etc.), responding to queries, performingspecific one-time tasks (such as responding to a voice instruction),performing ongoing tasks (such as schedule management and personalhealth management), and providing recommendations. By way of example,these tasks or services can be based on one or more of user input,prediction, activity awareness, location awareness, an ability to accessinformation (including user profile information and online information),user profile information, and other data or information.

Wearable electronic device 1508 includes computer readable medium (CRM)or memory 1570, one or more displays 1572, a processor or processingunit 1574, one or more interfaces 1576 (such as an interface discussedherein), a camera 1578, one or more sensors 1580 (such as a sensordiscussed herein), a motion or location tracker 1582, one or morespeakers 1584, one or more impulse responses, transfer functions, andSLPs 1586, a head tracking system or head tracker 1588, an imagerysystem 1590, a sound localization system 1592, and one or moremicrophones 1594.

The earphones 1510 include a left and a right speaker 1596 and a leftand a right microphone 1598.

By way of example, the sound hardware 1546 includes a sound card and/ora sound chip. A sound card includes one or more of a digital-to-analog(DAC) converter, an analog-to-digital (ATD) converter, a line-inconnector for an input signal from a sound source, a line-out connector,a hardware audio accelerator providing hardware polyphony, and adigital-signal-processor (DSP). A sound chip is an integrated circuit(also known as a “chip”) that produces sound through digital, analog, ormixed-mode electronics and includes electronic devices such as one ormore of an oscillator, envelope controller, sampler, filter, andamplifier.

By way of example, the imagery system 1590 includes, but is not limitedto, one or more of an optical projection system, a virtual image displaysystem, virtual augmented reality system, lenses, and/or a spatialaugmented reality system. By way of example, the virtual augmentedreality system uses one or more of image registration, computer vision,and/or video tracking to supplement and/or change real objects and/or aview of the physical, real world.

In some instances, a HPED may not have an internal speaker capable ofgenerating a sound sufficient to capture impulse responses at themicrophones. The HPED can communicate with a separate sound impulsegenerator (such as a separate speaker 1512 proximate to the HPED) andcause this impulse generator to emit the sound to generate the impulseresponses at the microphones. For example, the sound impulse generator1512 can be a speaker coupled with or in communication with the HPED andmounted together with the HPED. The HPED can trigger the sound impulsegenerator to make the sound. An orientation allows the location anddirection of transmission of the sound generated to be similar or thesame for both the HPED speaker and the sound impulse generator 1512.

The event predictor or predictor 1542 predicts or estimates eventsincluding, but not limited to, switching or changing between binauraland stereo sounds at a future time, changing or altering binaural sound(such as moving a SLP, reducing a number of SLPs, eliminating a SLP,adding a SLP, starting transmission or emission of binaural sound,stopping transmission or emission of binaural sound, etc.), predictingan action of a user, predicting a location of a user, predicting anevent, predicting a desire or want of a user, predicting a query of user(such as a query to an intelligent personal assistant), etc. Thepredictor can also predict user actions or requests in the future (suchas a likelihood that the user or electronic device requests a switchbetween binaural and stereo sounds or a change to binaural sound). Forinstance, determinations by a software application, an electronicdevice, and/or the user agent can be modeled as a prediction that theuser will take an action and/or desire or benefit from a switch betweenbinaural and stereo sounds or a change to binaural sound (such aspausing binaural sound, muting binaural sound, reducing or eliminatingone or more cues or spatializations or localizations of binaural sound).For example, an analysis of historic events, personal information,geographic location, and/or the user profile provides a probabilityand/or likelihood that the user will take an action (such as whether theuser prefers binaural sound or stereo sound for a particular location, aparticular listening experience, or a particular communication withanother person or an intelligent personal assistant). By way of example,one or more predictive models are used to predict the probability that auser would take, determine, or desire the action.

The predictive models can use one or more classifiers to determine theseprobabilities. Example models and/or classifiers include, but are notlimited to, a Naive Bayes classifier (including classifiers that applyBayes' theorem), k-nearest neighbor algorithm (k-NN, includingclassifying objects based on a closeness to training examples in featurespace), statistics (including the collection, organization, and analysisof data), collaborative filtering, support vector machine (SVM,including supervised learning models that analyze data and recognizepatterns in data), data mining (including discovery of patterns indata-sets), artificial intelligence (including systems that useintelligent agents to perceive environments and take action based on theperceptions), machine learning (including systems that learn from data),pattern recognition (including classification, regression, sequencelabeling, speech tagging, and parsing), knowledge discovery (includingthe creation and analysis of data from databases and unstructured datasources), logistic regression (including generation of predictions usingcontinuous and/or discrete variables), group method of data handling(GMDH, including inductive algorithms that model multi-parameter data)and uplift modeling (including analyzing and modeling changes inprobability due to an action).

Consider an example in which the predictor tracks and stores event dataover a period of time, such as days, weeks, months, or years for usersof binaural sound. This event data includes recording and analyzingpatterns of actions with the binaural sound and motions of an electronicdevice (such as an HPED or electronic earphones). Based on this historicinformation, the predictor predicts what action a particular user willtake with an electronic device (e.g., whether the user will accept orplace a voice call in binaural sound or stereo sound and with whom andat what time and locations, whether the user will communicate with anintelligent personal assistant in binaural sound or stereo sound at whattimes and locations and for what durations, whether the user will listento music in binaural sound or stereo sound and from which sources, wherethe user will take the electronic device, in what orientation it will becarried, the travel time to the destination and the route to get there,in what direction a user will walk or turn or orient his/her head orgaze, what mood or emotion a user is experiencing, etc.).

One or more electronic devices can also monitor and collect data withrespect to the person and/or electronic devices, such as electronicdevices that the person interacts with and/or owns. By way of example,this data includes user behavior on an electronic device, installedclient hardware, installed client software, locally stored client files,information obtained or generated from the user's interaction with anetwork (such as web pages on the internet), email, peripheral devices,servers, other electronic devices, programs that are executing, SLPlocations, SLP preferences, binaural sound preferences, music listeningpreferences, time of day and period of use, sensor readings (such ascommon gaze angles and patterns of gaze at certain locations such as awork desk or home armchair, common device orientations and cyclicalpatterns of orientation such as one gathered while a device is in apocket or on a head), etc. The electronic devices collect user behavioron or with respect to an electronic device (such as the user'scomputer), information about the user, information about the user'scomputer, and/or information about the computer's and/or user'sinteraction with the network.

By way of example, a user agent (including an IUA) and/or user profilebuilder monitors user activities and collects information used to createa user profile, and this user profile includes public and privateinformation. The profile builder monitors the user's interactions withone or more electronic devices, the user's interactions with othersoftware applications executing on electronic devices, activitiesperformed by the user on external or peripheral electronic devices, etc.The profile builder collects both content information and contextinformation for the monitored user activities and then stores thisinformation. By way of further illustration, the content informationincludes contents of web pages and internet links accessed by the user,people called, subjects spoken of, locations called, questions or tasksasked of an IPA, graphical information, audio/video information,patterns in head tracking, device orientation, location, physical andvirtual positions of conversations, searches or queries performed by theuser, items purchased, likes/dislikes of the user, advertisements viewedor clicked, information on commercial or financial transactions, videoswatched, music played, interactions between the user and a userinterface (UI) of an electronic device, commands (such as voice andtyped commands), information relating to SLPs and binaural sound, etc.

The user profile builder also gathers and stores information related tothe context in which the user performed activities associated with anelectronic device. By way of example, such context information includes,but is not limited to, an order, frequency, duration, and time of day inwhich the user accessed web pages, audio streams, SLPs, informationregarding the user's response to interactive advertisements, calls,requests and notifications from intelligent personal assistants (IPAs),information as to when or where a user localized binaural sounds,switched to or from binaural sound sending or receiving, etc.

As previously stated, the user profile builder also collects content andcontext information associated with the user interactions with variousdifferent applications executing on one or more electronic devices. Forexample, the user profile builder monitors and gathers data on theuser's interactions with a telephony application, an AAR application,web browser, an electronic mail (email) application, a word processorapplication, a spreadsheet application, a database application, a cloudsoftware application, a sound localization system (SLS), and/or anyother software application executing on an electronic device.

Consider an example in which a user agent and/or electronic devicegathers SLP preferences while the user communicates during a voiceexchange with an intelligent user agent, an intelligent personalassistant, or another person during a communication over the Internet.For example, a facial and emotional recognition system determines facialand body gestures of a user while the user communicates during the voiceexchange. For instance, this system can utilize Principal ComponentAnalysis with Eigenfaces, Linear Discriminate Analysis, 3D facialimaging techniques, emotion classification algorithms, BayesianReasoning, Support Vector Machines, K-Nearest Neighbor, neural networks,or a Hidden Markov Model. A machine learning classifier can be used torecognize an emotion of the user.

By way of example, SLP preferences can include a person's personal likesand dislikes, opinions, traits, recommendations, priorities, tastes,subjective information, etc. with regard to SLPs and binaural sound. Forinstance, the preferences include a desired or preferred location for aSLP during a voice exchange, a desired or preferred time when tolocalize sound versus not localize sound, permissions that grant or denypeople rights to localize to a SLP that is away from but proximate to aperson during a voice exchange (such as a VoIP call), a size and/orshape of a SLP, a length of time that sound localizes to a SLP, apriority of a SLP, a number of SLPs that simultaneously localize to aperson, etc. Consider an example in which a HPED has a mobile operatingsystem that includes a computer program that is an intelligent personalassistant (IPA) and knowledge navigator. The IPA uses a natural languageuser interface to interact with a user, answer questions, performservices, make recommendations, and communicates with a database and webservices to assist the user. The IPA further includes or communicateswith a predictor and/or user profiler to provide its user withindividualized searches and functions specific to and based onpreferences of the user. A conversational interface (e.g., using anatural language interface with voice recognition), personal contextawareness (e.g., using user profile data to adapt to individualpreferences with personalized results), and service delegation (e.g.,providing access to built-in applications in the HPED) enable the IPA tointeract with its user and perform functions discussed herein. Forexample, the IPA predicts and/or intelligently performs generating andcapturing the sound from a HPED to acquire the impulse responses and/ortransfer functions or and executing other methods discussed herein.

Consider an example in which a HPED has a mobile operating system with acomputer program that is an intelligent personal assistant (IPA) andknowledge navigator. The IPA uses a natural language user interface tointeract with a user, answer questions, perform services, makerecommendations, and communicate with a database and web services toassist the user. The IPA further includes or communicates with apredictor and/or user profile to provide its user with individualizedsearches and functions specific to and based on preferences of the user,such as selecting a SLP at a location. A conversational interface (e.g.,using a natural language interface with voice recognition and machinelearning), personal context awareness (e.g., using user profile data toadapt to individual preferences and provide personalized results), andservice delegation (e.g., providing access to built-in applications inthe HPED) enable the IPA to interact with its user and perform functionsdiscussed herein (such as one or more blocks in the figures). Forexample, the IPA predicts and/or intelligently performs generating atone for an impulse response, convolving sounds with specific impulseresponse or transfer functions, selecting between multiple SLPs at alocation, and executing other methods discussed herein.

FIGS. 14 and 15 show example electronic devices with various components.One or more of these components can be distributed or included invarious electronic devices, such as some components being included in anHPED, some components being included in a server, some components beingincluded in storage accessible over the Internet, some components beingin an imagery system, some components being in wearable electronicdevices, and some components being in various different electronicdevices that are spread across a network or a cloud, etc.

The speaker that generates the sound to capture the impulses responsescan be physically separate from the HPED. By way of example, FIG. 16shows an electronic system 1600 in which a HPED 1601 and an impulsegenerator or speaker 1603 connect to an end of a pole, rod,selfie-stick, or monopod 1604. The HPED 1601 and the speaker 1603 areconnected to or positioned between a back brace 1605 and a front brace1606 such that the HPED and speaker are next to each other and pointingin a same or similar direction for generation and capture of audioimpulse responses as discussed herein.

Example embodiments are not limited to using a HPED to generate a soundfrom its speaker to capture audio impulse responses since the HPED canexecute one or more blocks discussed herein to manage HRTFs in otherways. For example, the HPED executes one or more blocks discussed hereinand designates a location for a SLP, and retrieves HRTFs or HRIRs forthe location of this SLP. For example, Alice holds the HPED in her handaway from her face and provides a command to the HPED to capture a SLP(e.g., Alice issues a voice command or taps on the display). In responseto this command, the HPED determines its location with respect to theface or facial orientation of Alice and generates a SLP for thislocation (such as determining distance, azimuth angle and/or elevationangle). The HPED then retrieves an individualized HRTF for Alice frommemory (such as an online database) and convolves sound with this HRTFso the sound localizes to the SLP that coincides with where the HPED waswhen Alice provided the command to the HPED. The HPED can select a setof HRTFs for this location of the SLP or another location near this SLP.For example, Alice holds the HPED at a location for a near-field HRTF,but the HPED retrieves a far-field HRTF corresponding to the azimuth andelevation angles of where the SLP was generated.

Blocks and/or methods discussed herein can be executed and/or made by auser, a user agent (including machine learning agents and intelligentuser agents), a software application, an electronic device, a computer,firmware, hardware, a process, a computer system, and/or an intelligentpersonal assistant. Furthermore, blocks and/or methods discussed hereincan be executed automatically with or without instruction from a user.

As used herein, “impulse response” is a reaction to an audio inputsignal in response to external change. Impulse responses includerecordings of reverberation caused by an acoustic space (such as a room,human head and/or body, dummy head, etc.) when an impulse is played. Theimpulse response can be from physical objects or mathematical systems ofequations describing or estimating the objects. Further, the impulse canbe modeled in either discrete time or continuous time. Furthermore,systems can use transfer functions or impulse response for analysis (thetransfer function being a Laplace Transform of the impulse response).Examples of impulse responses include RIRs, HRIRs, and BRIRs.

As used herein, “line-of-sight” is the forward-looking direction of theface of the person that extends along a straight line from an end of thenose of the person.

As used herein, a “user” can be a human being, an intelligent personalassistant (IPA), a user agent (including an intelligent user agent and amachine learning agent), a process, a computer system, a server, asoftware program, hardware, an avatar, or an electronic device. A usercan also have a name, such as Alice, Bob, and Charlie, as described insome example embodiments.

As used herein, a “user agent” is software that acts on behalf of auser. User agents include, but are not limited to, one or more ofintelligent user agents and/or intelligent electronic personalassistants (IPAs, software agents, and/or assistants that use learning,reasoning and/or artificial intelligence), multi-agent systems (pluralagents that communicate with each other), mobile agents (agents thatmove execution to different processors), autonomous agents (agents thatmodify processes to achieve an objective), and distributed agents(agents that execute on physically distinct electronic devices).

As used herein, a “user profile” is personal data that represents anidentity of a specific person or organization. The user profile includesinformation pertaining to the characteristics and/or preferences of theuser. Examples of this information for a person include, but are notlimited to, one or more of personal data of the user (such as age,gender, race, ethnicity, religion, hobbies, interests, income,employment, education, location, communication hardware and softwareused including peripheral devices such as head tracking systems,abilities, disabilities, biometric data, physical measurements of theirbody and environments, functions of physical data such as HRTFs, etc.),photographs (such as photos of the user, family, friends, and/orcolleagues, their head and ears), videos (such as videos of the user,family, friends, and/or colleagues), and user-specific data that definesthe user's interaction with and/or content on an electronic device (suchas display settings, audio settings, application settings, networksettings, stored files, downloads/uploads, browser and calling activity,software applications, user interface or GUI activities, and/orprivileges).

Examples herein can take place in physical spaces, in computer renderedspaces (VR), in partially computer rendered spaces (AR), and incombinations thereof.

The processor unit includes a processor (such as a central processingunit, CPU, microprocessor, field programmable gate array (FPGA),application-specific integrated circuit (ASIC), etc.) for controllingthe overall operation of memory (such as random access memory (RAM) fortemporary data storage, read only memory (ROM) for permanent datastorage, and firmware). The processing unit communicates with memory andperforms operations and tasks that implement one or more blocks of theflow diagrams discussed herein. The memory, for example, storesapplications, data, programs, algorithms (including software toimplement or assist in implementing example embodiments) and other data.

Consider an example in which the SLS or portions of the SLS include anintegrated circuit FPGA that is specifically customized, designed,configured, or wired to execute one or more blocks discussed herein. Forexample, the FPGA includes one or more programmable logic blocks thatare wired together or configured to execute combinational functions forthe SLS.

Consider an example in which the SLS or portions of the SLS include anintegrated circuit or ASIC that is specifically customized, designed, orconfigured to execute one or more blocks discussed herein. For example,the ASIC has customized gate arrangements for the SLS. The ASIC can alsoinclude microprocessors and memory blocks (such as being a SoC(system-on-chip) designed with special functionality to executefunctions of the SLS).

Consider an example in which the SLS or portions of the SLS include oneor more integrated circuits that are specifically customized, designed,or configured to execute one or more blocks discussed herein.

In some example embodiments, the methods illustrated herein and data andinstructions associated therewith are stored in respective storagedevices, which are implemented as computer-readable and/ormachine-readable storage media, physical or tangible media, and/ornon-transitory storage media. These storage media include differentforms of memory including semiconductor memory devices such as DRAM, orSRAM, Erasable and Programmable Read-Only Memories (EPROMs),Electrically Erasable and Programmable Read-Only Memories (EEPROMs) andflash memories; magnetic disks such as fixed, floppy and removabledisks; other magnetic media including tape; optical media such asCompact Disks (CDs) or Digital Versatile Disks (DVDs). Note that theinstructions of the software discussed above can be provided oncomputer-readable or machine-readable storage medium, or alternatively,can be provided on multiple computer-readable or machine-readablestorage media distributed in a large system having possibly pluralnodes. Such computer-readable or machine-readable medium or media is(are) considered to be part of an article (or article of manufacture).An article or article of manufacture can refer to any manufacturedsingle component or multiple components.

Method blocks discussed herein can be automated and executed by acomputer, computer system, user agent, and/or electronic device. Theterm “automated” means controlled operation of an apparatus, system,and/or process using computers and/or mechanical/electrical deviceswithout the necessity of human intervention, observation, effort, and/ordecision.

The methods in accordance with example embodiments are provided asexamples, and examples from one method should not be construed to limitexamples from another method. Further, methods discussed withindifferent figures can be added to or exchanged with methods in otherfigures. Further yet, specific numerical data values (such as specificquantities, numbers, categories, etc.) or other specific informationshould be interpreted as illustrative for discussing exampleembodiments. Such specific information is not provided to limit exampleembodiments.

What is claimed is:
 1. A method comprising: dividing, with a portableelectronic device (PED) held in a hand of a user, an area around theuser into a zone that extends around the user and that includes a soundlocalization point (SLP) from where binaural sound in empty spaceoriginates to the user; playing, with speakers in a wearable electronicdevice (WED) worn on a head of the user, the binaural sound thatoriginates from the SLP in empty space; tracking, with one or moresensors in the WED worn on the head of the user, the PED to determinewhen the PED held in the hand of the user is moving outside the zonethat extends around the user; and displaying, with the WED worn on thehead of the user, a virtual reality (VR) image that shows the zone inresponse to the WED determining that the PED is moving outside the zone.2. The method of claim 1 further comprising: highlighting, while theuser is located inside the zone and with a display of the WED, a VRimage at the SLP in the zone when the PED held in the hand of the useris pointed at the SLP in the zone; and playing, with the WED worn on thehead of the user, the binaural sound that emanates from the SLP inresponse to the PED pointing at the SLP.
 3. The method of claim 1further comprising: displaying, with the WED worn on the head of theuser, the VR image that shows a size and a shape of the zone in responseto the WED determining that the PED is moving outside the zone.
 4. Themethod of claim 1 further comprising: storing, in memory of the WED, alocation of the zone, a size and a shape of the zone, and a location ofthe SLP in the zone from where the binaural sound originates to theuser; and retrieving, from the memory of the WED, the size and the shapeof the zone in response to the WED determining that the user is in thelocation of the zone.
 5. The method of claim 1 further comprising:designating, with the PED held in the hand of the user, locations ofmultiple SLPs in empty space in the zone from where the binaural soundoriginates to the user; and displaying, with the WED, VR images at thelocations of the multiple SLPs in empty space in the zone.
 6. The methodof claim 1 further comprising: selecting one of multiple SLPs in thezone when the PED held in the hand of the user is pointed at the one ofthe multiple SLPs in the zone; and displaying, with the WED worn on thehead of the user, a VR image at the one of the multiple SLPs in responseto the one of the multiple SLPs being pointed at by the PED held in thehand of the user.
 7. The method of claim 1, wherein the zone has one ofa three-dimensional (3D) shape of a cylinder and a 3D shape of arectangle, and the user is centered in the zone.
 8. The method of claim1 further comprising: playing, with the speakers in the WED worn on thehead of the user, an audio warning in response to the WED determiningthat the PED is moving outside the zone.
 9. A method comprising:dividing, with a portable electronic device (PED) held in a hand of auser, an area around the user into a three-dimensional (3D) zone thatextends around the user and that includes sound localization points(SLPs) from where binaural sound originates to the user; determining,with a wearable electronic device (WED) worn on a head of the user, whenthe user is leaving the zone; and warning the user that the user isleaving the zone by displaying, with the WED worn on the head of theuser, a 3D virtual reality (VR) image of the zone in response to the WEDdetermining that the user is leaving the zone.
 10. The method of claim 9further comprising: determining, with the WED, when the PED is pointedat one of the SLPs while the user is located in the zone; and playingthe binaural sound that originates from the one of the SLPs in the zonein response to the WED determining that the PED is pointed at the one ofthe SLPs.
 11. The method of claim 9 further comprising: highlighting,while the user is located in the zone and with a display of the WED,color of a VR image at one of the SLPs in the zone when the PED held inthe hand of the user points at the one of the SLPs in the zone.
 12. Themethod of claim 9, wherein the 3D VR image of the zone is shaped as acylinder with the user being centered in the cylinder, and the WED is ahead mounted display that display VR images and plays the binaural soundwhile the user plays a game.
 13. The method of claim 9, wherein the 3DVR image of the zone is shaped as a rectangle with the user beingcentered in the rectangle, and the WED is a head mounted display thatdisplays VR images and plays the binaural sound while the user plays agame.
 14. The method of claim 9 further comprising: playing, with theWED worn on the head of the user, an audio warning indicating the usershould move back into the zone in response to the WED determining thatthe user is leaving the zone.
 15. The method of claim 9, wherein the 3DVR image of the zone is shaped as a cylinder that extends around theuser.
 16. The method of claim 9 further comprising: determining, withthe WED worn on the head of the user, when the PED is leaving the zone;and warning the user that the PED is leaving the zone by displaying,with the WED worn on the head of the user, the 3D VR image of the zonein response to the WED determining that the PED is leaving the zone. 17.An electronic system comprising: a portable electronic device (PED) heldin a hand of a user that divides an area around the user into athree-dimensional (3D) zone that extends around the user and includessound localization points (SLPs) in empty space from where binauralsound originates to the user while the user is located inside the zone;and a wearable electronic device (WED) that is worn on a head of theuser, wirelessly communicates with the PED, includes speakers that playto the user the binaural sound that emanates from the SLPs in emptyspace inside the zone, includes one or more sensors that sense when thePED is leaving the zone, and includes a display that displays a virtualreality (VR) image of the zone in response to sensing that the PED isleaving the zone, wherein the display displays the VR image of the zoneto warn the user that the user is leaving the zone.
 18. The electronicsystem of claim 17, wherein the display of the WED highlights a color ofa VR image at one of the SLPs in response to the WED determining thatthe PED is pointing to the VR image at the one of the SLPs.
 19. Theelectronic system of claim 17, wherein the WED tracks a location of theuser with respect to the SLPs, and automatically displays a VR image ata location of one of the SLPs in response to determining that the useris proximate to the location of the one of the SLPs.
 20. The electronicsystem of claim 17, wherein the speakers in the WED play the binauralsound that emanates from one of the SLPs in empty space in response tothe WED determining that the PED is pointed to the one of the SLPs inempty space, and the zone is shaped as a 3D cylinder with the user beingcentered in the 3D cylinder.